cpython-lru_cache-weakref: Lib/test/test

annotate Lib/test/test_codecs.py @ 54294:dba767bc3155 legacy-trunk

remove mention of old ctypes version

author	Benjamin Peterson <benjamin@python.org>
date	Sat, 23 May 2009 20:59:09 +0000
parents	7dca250343ea
children	6768b2350c57

rev	line source
24364 aa0d2d51fbde Get rid of relative imports in all unittests. Now anything that Barry Warsaw <barry@python.org> parents: 20013 diff changeset	1 from test import test_support
aa0d2d51fbde Get rid of relative imports in all unittests. Now anything that Barry Warsaw <barry@python.org> parents: 20013 diff changeset	2 import unittest
18163 f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	3 import codecs
37077 38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	4 import sys, StringIO, _testcapi
18163 f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	5
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	6 class Queue(object):
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	7 """
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	8 queue: write bytes at one end, read bytes from the other end
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	9 """
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	10 def __init__(self):
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	11 self._buffer = ""
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	12
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	13 def write(self, chars):
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	14 self._buffer += chars
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	15
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	16 def read(self, size=-1):
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	17 if size<0:
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	18 s = self._buffer
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	19 self._buffer = ""
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	20 return s
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	21 else:
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	22 s = self._buffer[:size]
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	23 self._buffer = self._buffer[size:]
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	24 return s
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	25
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	26 class ReadTest(unittest.TestCase):
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	27 def check_partial(self, input, partialresults):
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	28 # get a StreamReader for the encoding and feed the bytestring version
43903 6aa8e135d9f8 Fix typo in comment. Walter Dörwald <walter@livinglogic.de> parents: 42649 diff changeset	29 # of input to the reader byte by byte. Read everything available from
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	30 # the StreamReader and check that the results equal the appropriate
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	31 # entries from partialresults.
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	32 q = Queue()
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	33 r = codecs.getreader(self.encoding)(q)
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	34 result = u""
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	35 for (c, partialresult) in zip(input.encode(self.encoding), partialresults):
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	36 q.write(c)
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	37 result += r.read()
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	38 self.assertEqual(result, partialresult)
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	39 # check that there's nothing left in the buffers
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	40 self.assertEqual(r.read(), u"")
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	41 self.assertEqual(r.bytebuffer, "")
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	42 self.assertEqual(r.charbuffer, u"")
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	43
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	44 # do the check again, this time using a incremental decoder
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	45 d = codecs.getincrementaldecoder(self.encoding)()
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	46 result = u""
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	47 for (c, partialresult) in zip(input.encode(self.encoding), partialresults):
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	48 result += d.decode(c)
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	49 self.assertEqual(result, partialresult)
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	50 # check that there's nothing left in the buffers
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	51 self.assertEqual(d.decode("", True), u"")
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	52 self.assertEqual(d.buffer, "")
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	53
43932 e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls. Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 43905 diff changeset	54 # Check whether the reset method works properly
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	55 d.reset()
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	56 result = u""
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	57 for (c, partialresult) in zip(input.encode(self.encoding), partialresults):
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	58 result += d.decode(c)
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	59 self.assertEqual(result, partialresult)
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	60 # check that there's nothing left in the buffers
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	61 self.assertEqual(d.decode("", True), u"")
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	62 self.assertEqual(d.buffer, "")
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	63
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	64 # check iterdecode()
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	65 encoded = input.encode(self.encoding)
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	66 self.assertEqual(
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	67 input,
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	68 u"".join(codecs.iterdecode(encoded, self.encoding))
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	69 )
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	70
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	71 def test_readline(self):
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	72 def getreader(input):
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	73 stream = StringIO.StringIO(input.encode(self.encoding))
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	74 return codecs.getreader(self.encoding)(stream)
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	75
36855 204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	76 def readalllines(input, keepends=True, size=None):
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	77 reader = getreader(input)
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	78 lines = []
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	79 while True:
36855 204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	80 line = reader.readline(size=size, keepends=keepends)
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	81 if not line:
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	82 break
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	83 lines.append(line)
36855 204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	84 return "\|".join(lines)
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	85
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	86 s = u"foo\nbar\r\nbaz\rspam\u2028eggs"
36855 204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	87 sexpected = u"foo\n\|bar\r\n\|baz\r\|spam\u2028\|eggs"
204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	88 sexpectednoends = u"foo\|bar\|baz\|spam\|eggs"
204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	89 self.assertEqual(readalllines(s, True), sexpected)
204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	90 self.assertEqual(readalllines(s, False), sexpectednoends)
204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	91 self.assertEqual(readalllines(s, True, 10), sexpected)
204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	92 self.assertEqual(readalllines(s, False, 10), sexpectednoends)
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	93
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	94 # Test long lines (multiple calls to read() in readline())
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	95 vw = []
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	96 vwo = []
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	97 for (i, lineend) in enumerate(u"\n \r\n \r \u2028".split()):
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	98 vw.append((i200)u"\3042" + lineend)
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	99 vwo.append((i200)u"\3042")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	100 self.assertEqual(readalllines("".join(vw), True), "".join(vw))
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	101 self.assertEqual(readalllines("".join(vw), False),"".join(vwo))
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	102
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	103 # Test lines where the first read might end with \r, so the
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	104 # reader has to look ahead whether this is a lone \r or a \r\n
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	105 for size in xrange(80):
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	106 for lineend in u"\n \r\n \r \u2028".split():
34865 b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	107 s = 10(sizeu"a" + lineend + u"xxx\n")
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	108 reader = getreader(s)
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	109 for i in xrange(10):
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	110 self.assertEqual(
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	111 reader.readline(keepends=True),
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	112 size*u"a" + lineend,
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	113 )
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	114 reader = getreader(s)
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	115 for i in xrange(10):
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	116 self.assertEqual(
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	117 reader.readline(keepends=False),
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	118 size*u"a",
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	119 )
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	120
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	121 def test_bug1175396(self):
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	122 s = [
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	123 '<%!--===================================================\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	124 ' BLOG index page: show recent articles,\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	125 ' today\'s articles, or articles of a specific date.\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	126 '========================================================--%>\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	127 '<%@inputencoding="ISO-8859-1"%>\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	128 '<%@pagetemplate=TEMPLATE.y%>\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	129 '<%@import=import frog.util, frog%>\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	130 '<%@import=import frog.objects%>\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	131 '<%@import=from frog.storageerrors import StorageError%>\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	132 '<%\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	133 '\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	134 'import logging\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	135 'log=logging.getLogger("Snakelets.logger")\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	136 '\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	137 '\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	138 'user=self.SessionCtx.user\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	139 'storageEngine=self.SessionCtx.storageEngine\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	140 '\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	141 '\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	142 'def readArticlesFromDate(date, count=None):\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	143 ' entryids=storageEngine.listBlogEntries(date)\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	144 ' entryids.reverse() # descending\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	145 ' if count:\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	146 ' entryids=entryids[:count]\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	147 ' try:\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	148 ' return [ frog.objects.BlogEntry.load(storageEngine, date, Id) for Id in entryids ]\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	149 ' except StorageError,x:\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	150 ' log.error("Error loading articles: "+str(x))\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	151 ' self.abort("cannot load articles")\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	152 '\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	153 'showdate=None\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	154 '\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	155 'arg=self.Request.getArg()\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	156 'if arg=="today":\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	157 ' #-------------------- TODAY\'S ARTICLES\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	158 ' self.write("<h2>Today\'s articles</h2>")\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	159 ' showdate = frog.util.isodatestr() \r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	160 ' entries = readArticlesFromDate(showdate)\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	161 'elif arg=="active":\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	162 ' #-------------------- ACTIVE ARTICLES redirect\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	163 ' self.Yredirect("active.y")\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	164 'elif arg=="login":\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	165 ' #-------------------- LOGIN PAGE redirect\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	166 ' self.Yredirect("login.y")\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	167 'elif arg=="date":\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	168 ' #-------------------- ARTICLES OF A SPECIFIC DATE\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	169 ' showdate = self.Request.getParameter("date")\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	170 ' self.write("<h2>Articles written on %s</h2>"% frog.util.mediumdatestr(showdate))\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	171 ' entries = readArticlesFromDate(showdate)\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	172 'else:\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	173 ' #-------------------- RECENT ARTICLES\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	174 ' self.write("<h2>Recent articles</h2>")\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	175 ' dates=storageEngine.listBlogEntryDates()\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	176 ' if dates:\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	177 ' entries=[]\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	178 ' SHOWAMOUNT=10\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	179 ' for showdate in dates:\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	180 ' entries.extend( readArticlesFromDate(showdate, SHOWAMOUNT-len(entries)) )\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	181 ' if len(entries)>=SHOWAMOUNT:\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	182 ' break\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	183 ' \r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	184 ]
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	185 stream = StringIO.StringIO("".join(s).encode(self.encoding))
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	186 reader = codecs.getreader(self.encoding)(stream)
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	187 for (i, line) in enumerate(reader):
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	188 self.assertEqual(line, s[i])
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	189
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	190 def test_readlinequeue(self):
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	191 q = Queue()
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	192 writer = codecs.getwriter(self.encoding)(q)
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	193 reader = codecs.getreader(self.encoding)(q)
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	194
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	195 # No lineends
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	196 writer.write(u"foo\r")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	197 self.assertEqual(reader.readline(keepends=False), u"foo")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	198 writer.write(u"\nbar\r")
34925 1331c7799640 Update test to the current readline() behaviour. Walter Dörwald <walter@livinglogic.de> parents: 34865 diff changeset	199 self.assertEqual(reader.readline(keepends=False), u"")
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	200 self.assertEqual(reader.readline(keepends=False), u"bar")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	201 writer.write(u"baz")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	202 self.assertEqual(reader.readline(keepends=False), u"baz")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	203 self.assertEqual(reader.readline(keepends=False), u"")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	204
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	205 # Lineends
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	206 writer.write(u"foo\r")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	207 self.assertEqual(reader.readline(keepends=True), u"foo\r")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	208 writer.write(u"\nbar\r")
34925 1331c7799640 Update test to the current readline() behaviour. Walter Dörwald <walter@livinglogic.de> parents: 34865 diff changeset	209 self.assertEqual(reader.readline(keepends=True), u"\n")
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	210 self.assertEqual(reader.readline(keepends=True), u"bar\r")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	211 writer.write(u"baz")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	212 self.assertEqual(reader.readline(keepends=True), u"baz")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	213 self.assertEqual(reader.readline(keepends=True), u"")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	214 writer.write(u"foo\r\n")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	215 self.assertEqual(reader.readline(keepends=True), u"foo\r\n")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	216
34384 a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	217 def test_bug1098990_a(self):
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	218 s1 = u"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy\r\n"
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	219 s2 = u"offending line: ladfj askldfj klasdj fskla dfzaskdj fasklfj laskd fjasklfzzzzaa%whereisthis!!!\r\n"
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	220 s3 = u"next line.\r\n"
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	221
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	222 s = (s1+s2+s3).encode(self.encoding)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	223 stream = StringIO.StringIO(s)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	224 reader = codecs.getreader(self.encoding)(stream)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	225 self.assertEqual(reader.readline(), s1)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	226 self.assertEqual(reader.readline(), s2)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	227 self.assertEqual(reader.readline(), s3)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	228 self.assertEqual(reader.readline(), u"")
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	229
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	230 def test_bug1098990_b(self):
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	231 s1 = u"aaaaaaaaaaaaaaaaaaaaaaaa\r\n"
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	232 s2 = u"bbbbbbbbbbbbbbbbbbbbbbbb\r\n"
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	233 s3 = u"stillokay:bbbbxx\r\n"
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	234 s4 = u"broken!!!!badbad\r\n"
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	235 s5 = u"againokay.\r\n"
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	236
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	237 s = (s1+s2+s3+s4+s5).encode(self.encoding)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	238 stream = StringIO.StringIO(s)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	239 reader = codecs.getreader(self.encoding)(stream)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	240 self.assertEqual(reader.readline(), s1)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	241 self.assertEqual(reader.readline(), s2)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	242 self.assertEqual(reader.readline(), s3)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	243 self.assertEqual(reader.readline(), s4)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	244 self.assertEqual(reader.readline(), s5)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	245 self.assertEqual(reader.readline(), u"")
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	246
42649 c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	247 class UTF32Test(ReadTest):
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	248 encoding = "utf-32"
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	249
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	250 spamle = ('\xff\xfe\x00\x00'
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	251 's\x00\x00\x00p\x00\x00\x00a\x00\x00\x00m\x00\x00\x00'
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	252 's\x00\x00\x00p\x00\x00\x00a\x00\x00\x00m\x00\x00\x00')
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	253 spambe = ('\x00\x00\xfe\xff'
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	254 '\x00\x00\x00s\x00\x00\x00p\x00\x00\x00a\x00\x00\x00m'
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	255 '\x00\x00\x00s\x00\x00\x00p\x00\x00\x00a\x00\x00\x00m')
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	256
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	257 def test_only_one_bom(self):
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	258 _,_,reader,writer = codecs.lookup(self.encoding)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	259 # encode some stream
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	260 s = StringIO.StringIO()
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	261 f = writer(s)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	262 f.write(u"spam")
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	263 f.write(u"spam")
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	264 d = s.getvalue()
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	265 # check whether there is exactly one BOM in it
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	266 self.assert_(d == self.spamle or d == self.spambe)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	267 # try to read it back
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	268 s = StringIO.StringIO(d)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	269 f = reader(s)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	270 self.assertEquals(f.read(), u"spamspam")
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	271
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	272 def test_badbom(self):
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	273 s = StringIO.StringIO(4*"\xff")
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	274 f = codecs.getreader(self.encoding)(s)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	275 self.assertRaises(UnicodeError, f.read)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	276
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	277 s = StringIO.StringIO(8*"\xff")
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	278 f = codecs.getreader(self.encoding)(s)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	279 self.assertRaises(UnicodeError, f.read)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	280
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	281 def test_partial(self):
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	282 self.check_partial(
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	283 u"\x00\xff\u0100\uffff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	284 [
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	285 u"", # first byte of BOM read
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	286 u"", # second byte of BOM read
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	287 u"", # third byte of BOM read
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	288 u"", # fourth byte of BOM read => byteorder known
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	289 u"",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	290 u"",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	291 u"",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	292 u"\x00",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	293 u"\x00",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	294 u"\x00",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	295 u"\x00",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	296 u"\x00\xff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	297 u"\x00\xff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	298 u"\x00\xff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	299 u"\x00\xff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	300 u"\x00\xff\u0100",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	301 u"\x00\xff\u0100",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	302 u"\x00\xff\u0100",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	303 u"\x00\xff\u0100",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	304 u"\x00\xff\u0100\uffff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	305 ]
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	306 )
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	307
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	308 def test_errors(self):
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	309 self.assertRaises(UnicodeDecodeError, codecs.utf_32_decode,
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	310 "\xff", "strict", True)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	311
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	312 class UTF32LETest(ReadTest):
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	313 encoding = "utf-32-le"
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	314
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	315 def test_partial(self):
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	316 self.check_partial(
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	317 u"\x00\xff\u0100\uffff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	318 [
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	319 u"",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	320 u"",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	321 u"",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	322 u"\x00",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	323 u"\x00",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	324 u"\x00",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	325 u"\x00",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	326 u"\x00\xff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	327 u"\x00\xff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	328 u"\x00\xff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	329 u"\x00\xff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	330 u"\x00\xff\u0100",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	331 u"\x00\xff\u0100",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	332 u"\x00\xff\u0100",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	333 u"\x00\xff\u0100",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	334 u"\x00\xff\u0100\uffff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	335 ]
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	336 )
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	337
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	338 def test_simple(self):
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	339 self.assertEqual(u"\U00010203".encode(self.encoding), "\x03\x02\x01\x00")
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	340
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	341 def test_errors(self):
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	342 self.assertRaises(UnicodeDecodeError, codecs.utf_32_le_decode,
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	343 "\xff", "strict", True)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	344
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	345 class UTF32BETest(ReadTest):
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	346 encoding = "utf-32-be"
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	347
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	348 def test_partial(self):
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	349 self.check_partial(
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	350 u"\x00\xff\u0100\uffff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	351 [
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	352 u"",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	353 u"",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	354 u"",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	355 u"\x00",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	356 u"\x00",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	357 u"\x00",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	358 u"\x00",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	359 u"\x00\xff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	360 u"\x00\xff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	361 u"\x00\xff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	362 u"\x00\xff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	363 u"\x00\xff\u0100",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	364 u"\x00\xff\u0100",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	365 u"\x00\xff\u0100",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	366 u"\x00\xff\u0100",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	367 u"\x00\xff\u0100\uffff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	368 ]
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	369 )
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	370
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	371 def test_simple(self):
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	372 self.assertEqual(u"\U00010203".encode(self.encoding), "\x00\x01\x02\x03")
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	373
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	374 def test_errors(self):
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	375 self.assertRaises(UnicodeDecodeError, codecs.utf_32_be_decode,
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	376 "\xff", "strict", True)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	377
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	378 class UTF16Test(ReadTest):
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	379 encoding = "utf-16"
18163 f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	380
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	381 spamle = '\xff\xfes\x00p\x00a\x00m\x00s\x00p\x00a\x00m\x00'
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	382 spambe = '\xfe\xff\x00s\x00p\x00a\x00m\x00s\x00p\x00a\x00m'
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	383
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	384 def test_only_one_bom(self):
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	385 _,_,reader,writer = codecs.lookup(self.encoding)
18163 f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	386 # encode some stream
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	387 s = StringIO.StringIO()
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	388 f = writer(s)
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	389 f.write(u"spam")
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	390 f.write(u"spam")
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	391 d = s.getvalue()
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	392 # check whether there is exactly one BOM in it
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	393 self.assert_(d == self.spamle or d == self.spambe)
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	394 # try to read it back
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	395 s = StringIO.StringIO(d)
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	396 f = reader(s)
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	397 self.assertEquals(f.read(), u"spamspam")
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	398
34538 37b2f11c1a00 Add a test for UTF-16 reading where the byte sequence doesn't start with Walter Dörwald <walter@livinglogic.de> parents: 34384 diff changeset	399 def test_badbom(self):
37b2f11c1a00 Add a test for UTF-16 reading where the byte sequence doesn't start with Walter Dörwald <walter@livinglogic.de> parents: 34384 diff changeset	400 s = StringIO.StringIO("\xff\xff")
34562 7a1277fad04e Fix stupid typo: Don't read from a writer. Walter Dörwald <walter@livinglogic.de> parents: 34538 diff changeset	401 f = codecs.getreader(self.encoding)(s)
34538 37b2f11c1a00 Add a test for UTF-16 reading where the byte sequence doesn't start with Walter Dörwald <walter@livinglogic.de> parents: 34384 diff changeset	402 self.assertRaises(UnicodeError, f.read)
37b2f11c1a00 Add a test for UTF-16 reading where the byte sequence doesn't start with Walter Dörwald <walter@livinglogic.de> parents: 34384 diff changeset	403
37b2f11c1a00 Add a test for UTF-16 reading where the byte sequence doesn't start with Walter Dörwald <walter@livinglogic.de> parents: 34384 diff changeset	404 s = StringIO.StringIO("\xff\xff\xff\xff")
34562 7a1277fad04e Fix stupid typo: Don't read from a writer. Walter Dörwald <walter@livinglogic.de> parents: 34538 diff changeset	405 f = codecs.getreader(self.encoding)(s)
34538 37b2f11c1a00 Add a test for UTF-16 reading where the byte sequence doesn't start with Walter Dörwald <walter@livinglogic.de> parents: 34384 diff changeset	406 self.assertRaises(UnicodeError, f.read)
37b2f11c1a00 Add a test for UTF-16 reading where the byte sequence doesn't start with Walter Dörwald <walter@livinglogic.de> parents: 34384 diff changeset	407
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	408 def test_partial(self):
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	409 self.check_partial(
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	410 u"\x00\xff\u0100\uffff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	411 [
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	412 u"", # first byte of BOM read
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	413 u"", # second byte of BOM read => byteorder known
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	414 u"",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	415 u"\x00",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	416 u"\x00",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	417 u"\x00\xff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	418 u"\x00\xff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	419 u"\x00\xff\u0100",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	420 u"\x00\xff\u0100",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	421 u"\x00\xff\u0100\uffff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	422 ]
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	423 )
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	424
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	425 def test_errors(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	426 self.assertRaises(UnicodeDecodeError, codecs.utf_16_decode, "\xff", "strict", True)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	427
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	428 class UTF16LETest(ReadTest):
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	429 encoding = "utf-16-le"
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	430
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	431 def test_partial(self):
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	432 self.check_partial(
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	433 u"\x00\xff\u0100\uffff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	434 [
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	435 u"",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	436 u"\x00",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	437 u"\x00",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	438 u"\x00\xff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	439 u"\x00\xff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	440 u"\x00\xff\u0100",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	441 u"\x00\xff\u0100",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	442 u"\x00\xff\u0100\uffff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	443 ]
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	444 )
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	445
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	446 def test_errors(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	447 self.assertRaises(UnicodeDecodeError, codecs.utf_16_le_decode, "\xff", "strict", True)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	448
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	449 class UTF16BETest(ReadTest):
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	450 encoding = "utf-16-be"
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	451
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	452 def test_partial(self):
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	453 self.check_partial(
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	454 u"\x00\xff\u0100\uffff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	455 [
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	456 u"",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	457 u"\x00",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	458 u"\x00",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	459 u"\x00\xff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	460 u"\x00\xff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	461 u"\x00\xff\u0100",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	462 u"\x00\xff\u0100",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	463 u"\x00\xff\u0100\uffff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	464 ]
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	465 )
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	466
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	467 def test_errors(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	468 self.assertRaises(UnicodeDecodeError, codecs.utf_16_be_decode, "\xff", "strict", True)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	469
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	470 class UTF8Test(ReadTest):
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	471 encoding = "utf-8"
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	472
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	473 def test_partial(self):
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	474 self.check_partial(
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	475 u"\x00\xff\u07ff\u0800\uffff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	476 [
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	477 u"\x00",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	478 u"\x00",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	479 u"\x00\xff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	480 u"\x00\xff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	481 u"\x00\xff\u07ff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	482 u"\x00\xff\u07ff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	483 u"\x00\xff\u07ff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	484 u"\x00\xff\u07ff\u0800",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	485 u"\x00\xff\u07ff\u0800",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	486 u"\x00\xff\u07ff\u0800",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	487 u"\x00\xff\u07ff\u0800\uffff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	488 ]
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	489 )
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	490
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	491 class UTF7Test(ReadTest):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	492 encoding = "utf-7"
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	493
43932 e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls. Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 43905 diff changeset	494 def test_partial(self):
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls. Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 43905 diff changeset	495 self.check_partial(
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls. Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 43905 diff changeset	496 u"a+-b",
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls. Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 43905 diff changeset	497 [
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls. Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 43905 diff changeset	498 u"a",
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls. Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 43905 diff changeset	499 u"a",
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls. Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 43905 diff changeset	500 u"a+",
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls. Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 43905 diff changeset	501 u"a+-",
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls. Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 43905 diff changeset	502 u"a+-b",
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls. Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 43905 diff changeset	503 ]
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls. Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 43905 diff changeset	504 )
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	505
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	506 class UTF16ExTest(unittest.TestCase):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	507
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	508 def test_errors(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	509 self.assertRaises(UnicodeDecodeError, codecs.utf_16_ex_decode, "\xff", "strict", 0, True)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	510
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	511 def test_bad_args(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	512 self.assertRaises(TypeError, codecs.utf_16_ex_decode)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	513
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	514 class ReadBufferTest(unittest.TestCase):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	515
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	516 def test_array(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	517 import array
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	518 self.assertEqual(
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	519 codecs.readbuffer_encode(array.array("c", "spam")),
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	520 ("spam", 4)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	521 )
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	522
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	523 def test_empty(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	524 self.assertEqual(codecs.readbuffer_encode(""), ("", 0))
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	525
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	526 def test_bad_args(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	527 self.assertRaises(TypeError, codecs.readbuffer_encode)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	528 self.assertRaises(TypeError, codecs.readbuffer_encode, 42)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	529
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	530 class CharBufferTest(unittest.TestCase):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	531
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	532 def test_string(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	533 self.assertEqual(codecs.charbuffer_encode("spam"), ("spam", 4))
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	534
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	535 def test_empty(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	536 self.assertEqual(codecs.charbuffer_encode(""), ("", 0))
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	537
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	538 def test_bad_args(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	539 self.assertRaises(TypeError, codecs.charbuffer_encode)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	540 self.assertRaises(TypeError, codecs.charbuffer_encode, 42)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	541
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	542 class UTF8SigTest(ReadTest):
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	543 encoding = "utf-8-sig"
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	544
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	545 def test_partial(self):
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	546 self.check_partial(
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	547 u"\ufeff\x00\xff\u07ff\u0800\uffff",
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	548 [
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	549 u"",
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	550 u"",
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	551 u"", # First BOM has been read and skipped
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	552 u"",
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	553 u"",
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	554 u"\ufeff", # Second BOM has been read and emitted
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	555 u"\ufeff\x00", # "\x00" read and emitted
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	556 u"\ufeff\x00", # First byte of encoded u"\xff" read
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	557 u"\ufeff\x00\xff", # Second byte of encoded u"\xff" read
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	558 u"\ufeff\x00\xff", # First byte of encoded u"\u07ff" read
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	559 u"\ufeff\x00\xff\u07ff", # Second byte of encoded u"\u07ff" read
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	560 u"\ufeff\x00\xff\u07ff",
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	561 u"\ufeff\x00\xff\u07ff",
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	562 u"\ufeff\x00\xff\u07ff\u0800",
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	563 u"\ufeff\x00\xff\u07ff\u0800",
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	564 u"\ufeff\x00\xff\u07ff\u0800",
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	565 u"\ufeff\x00\xff\u07ff\u0800\uffff",
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	566 ]
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	567 )
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	568
40386 a3e0f1e11082 Change decode() so that it works with a buffer (i.e. unicode(..., 'utf-8-sig')) Walter Dörwald <walter@livinglogic.de> parents: 40284 diff changeset	569 def test_bug1601501(self):
a3e0f1e11082 Change decode() so that it works with a buffer (i.e. unicode(..., 'utf-8-sig')) Walter Dörwald <walter@livinglogic.de> parents: 40284 diff changeset	570 # SF bug #1601501: check that the codec works with a buffer
a3e0f1e11082 Change decode() so that it works with a buffer (i.e. unicode(..., 'utf-8-sig')) Walter Dörwald <walter@livinglogic.de> parents: 40284 diff changeset	571 unicode("\xef\xbb\xbf", "utf-8-sig")
a3e0f1e11082 Change decode() so that it works with a buffer (i.e. unicode(..., 'utf-8-sig')) Walter Dörwald <walter@livinglogic.de> parents: 40284 diff changeset	572
41445 8447314f500a Fix utf-8-sig incremental decoder, which didn't recognise a BOM when the Walter Dörwald <walter@livinglogic.de> parents: 40386 diff changeset	573 def test_bom(self):
8447314f500a Fix utf-8-sig incremental decoder, which didn't recognise a BOM when the Walter Dörwald <walter@livinglogic.de> parents: 40386 diff changeset	574 d = codecs.getincrementaldecoder("utf-8-sig")()
8447314f500a Fix utf-8-sig incremental decoder, which didn't recognise a BOM when the Walter Dörwald <walter@livinglogic.de> parents: 40386 diff changeset	575 s = u"spam"
8447314f500a Fix utf-8-sig incremental decoder, which didn't recognise a BOM when the Walter Dörwald <walter@livinglogic.de> parents: 40386 diff changeset	576 self.assertEqual(d.decode(s.encode("utf-8-sig")), s)
8447314f500a Fix utf-8-sig incremental decoder, which didn't recognise a BOM when the Walter Dörwald <walter@livinglogic.de> parents: 40386 diff changeset	577
43905 0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	578 def test_stream_bom(self):
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	579 unistring = u"ABC\u00A1\u2200XYZ"
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	580 bytestring = codecs.BOM_UTF8 + "ABC\xC2\xA1\xE2\x88\x80XYZ"
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	581
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	582 reader = codecs.getreader("utf-8-sig")
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	583 for sizehint in [None] + range(1, 11) + \
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	584 [64, 128, 256, 512, 1024]:
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	585 istream = reader(StringIO.StringIO(bytestring))
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	586 ostream = StringIO.StringIO()
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	587 while 1:
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	588 if sizehint is not None:
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	589 data = istream.read(sizehint)
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	590 else:
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	591 data = istream.read()
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	592
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	593 if not data:
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	594 break
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	595 ostream.write(data)
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	596
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	597 got = ostream.getvalue()
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	598 self.assertEqual(got, unistring)
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	599
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	600 def test_stream_bare(self):
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	601 unistring = u"ABC\u00A1\u2200XYZ"
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	602 bytestring = "ABC\xC2\xA1\xE2\x88\x80XYZ"
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	603
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	604 reader = codecs.getreader("utf-8-sig")
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	605 for sizehint in [None] + range(1, 11) + \
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	606 [64, 128, 256, 512, 1024]:
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	607 istream = reader(StringIO.StringIO(bytestring))
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	608 ostream = StringIO.StringIO()
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	609 while 1:
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	610 if sizehint is not None:
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	611 data = istream.read(sizehint)
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	612 else:
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	613 data = istream.read()
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	614
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	615 if not data:
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	616 break
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	617 ostream.write(data)
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	618
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	619 got = ostream.getvalue()
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	620 self.assertEqual(got, unistring)
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	621
25100 bc590beadf0b Check whether a string resize is necessary at the end Walter Dörwald <walter@livinglogic.de> parents: 24364 diff changeset	622 class EscapeDecodeTest(unittest.TestCase):
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	623 def test_empty(self):
25100 bc590beadf0b Check whether a string resize is necessary at the end Walter Dörwald <walter@livinglogic.de> parents: 24364 diff changeset	624 self.assertEquals(codecs.escape_decode(""), ("", 0))
bc590beadf0b Check whether a string resize is necessary at the end Walter Dörwald <walter@livinglogic.de> parents: 24364 diff changeset	625
27680 b19aaaeb6d64 Fix for [ 543344 ] Interpreter crashes when recoding; suggested Marc-André Lemburg <mal@egenix.com> parents: 25100 diff changeset	626 class RecodingTest(unittest.TestCase):
b19aaaeb6d64 Fix for [ 543344 ] Interpreter crashes when recoding; suggested Marc-André Lemburg <mal@egenix.com> parents: 25100 diff changeset	627 def test_recoding(self):
b19aaaeb6d64 Fix for [ 543344 ] Interpreter crashes when recoding; suggested Marc-André Lemburg <mal@egenix.com> parents: 25100 diff changeset	628 f = StringIO.StringIO()
b19aaaeb6d64 Fix for [ 543344 ] Interpreter crashes when recoding; suggested Marc-André Lemburg <mal@egenix.com> parents: 25100 diff changeset	629 f2 = codecs.EncodedFile(f, "unicode_internal", "utf-8")
b19aaaeb6d64 Fix for [ 543344 ] Interpreter crashes when recoding; suggested Marc-André Lemburg <mal@egenix.com> parents: 25100 diff changeset	630 f2.write(u"a")
b19aaaeb6d64 Fix for [ 543344 ] Interpreter crashes when recoding; suggested Marc-André Lemburg <mal@egenix.com> parents: 25100 diff changeset	631 f2.close()
b19aaaeb6d64 Fix for [ 543344 ] Interpreter crashes when recoding; suggested Marc-André Lemburg <mal@egenix.com> parents: 25100 diff changeset	632 # Python used to crash on this at exit because of a refcount
b19aaaeb6d64 Fix for [ 543344 ] Interpreter crashes when recoding; suggested Marc-André Lemburg <mal@egenix.com> parents: 25100 diff changeset	633 # bug in _codecsmodule.c
20013 e3184e46303b Change the PyUnit-based tests to use the test_main() approach. This Fred Drake <fdrake@acm.org> parents: 18163 diff changeset	634
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	635 # From RFC 3492
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	636 punycode_testcases = [
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	637 # A Arabic (Egyptian):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	638 (u"\u0644\u064A\u0647\u0645\u0627\u0628\u062A\u0643\u0644"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	639 u"\u0645\u0648\u0634\u0639\u0631\u0628\u064A\u061F",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	640 "egbpdaj6bu4bxfgehfvwxn"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	641 # B Chinese (simplified):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	642 (u"\u4ED6\u4EEC\u4E3A\u4EC0\u4E48\u4E0D\u8BF4\u4E2D\u6587",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	643 "ihqwcrb4cv8a8dqg056pqjye"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	644 # C Chinese (traditional):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	645 (u"\u4ED6\u5011\u7232\u4EC0\u9EBD\u4E0D\u8AAA\u4E2D\u6587",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	646 "ihqwctvzc91f659drss3x8bo0yb"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	647 # D Czech: Pro<ccaron>prost<ecaron>nemluv<iacute><ccaron>esky
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	648 (u"\u0050\u0072\u006F\u010D\u0070\u0072\u006F\u0073\u0074"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	649 u"\u011B\u006E\u0065\u006D\u006C\u0075\u0076\u00ED\u010D"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	650 u"\u0065\u0073\u006B\u0079",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	651 "Proprostnemluvesky-uyb24dma41a"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	652 # E Hebrew:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	653 (u"\u05DC\u05DE\u05D4\u05D4\u05DD\u05E4\u05E9\u05D5\u05D8"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	654 u"\u05DC\u05D0\u05DE\u05D3\u05D1\u05E8\u05D9\u05DD\u05E2"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	655 u"\u05D1\u05E8\u05D9\u05EA",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	656 "4dbcagdahymbxekheh6e0a7fei0b"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	657 # F Hindi (Devanagari):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	658 (u"\u092F\u0939\u0932\u094B\u0917\u0939\u093F\u0928\u094D"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	659 u"\u0926\u0940\u0915\u094D\u092F\u094B\u0902\u0928\u0939"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	660 u"\u0940\u0902\u092C\u094B\u0932\u0938\u0915\u0924\u0947"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	661 u"\u0939\u0948\u0902",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	662 "i1baa7eci9glrd9b2ae1bj0hfcgg6iyaf8o0a1dig0cd"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	663
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	664 #(G) Japanese (kanji and hiragana):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	665 (u"\u306A\u305C\u307F\u3093\u306A\u65E5\u672C\u8A9E\u3092"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	666 u"\u8A71\u3057\u3066\u304F\u308C\u306A\u3044\u306E\u304B",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	667 "n8jok5ay5dzabd5bym9f0cm5685rrjetr6pdxa"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	668
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	669 # (H) Korean (Hangul syllables):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	670 (u"\uC138\uACC4\uC758\uBAA8\uB4E0\uC0AC\uB78C\uB4E4\uC774"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	671 u"\uD55C\uAD6D\uC5B4\uB97C\uC774\uD574\uD55C\uB2E4\uBA74"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	672 u"\uC5BC\uB9C8\uB098\uC88B\uC744\uAE4C",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	673 "989aomsvi5e83db1d2a355cv1e0vak1dwrv93d5xbh15a0dt30a5j"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	674 "psd879ccm6fea98c"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	675
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	676 # (I) Russian (Cyrillic):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	677 (u"\u043F\u043E\u0447\u0435\u043C\u0443\u0436\u0435\u043E"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	678 u"\u043D\u0438\u043D\u0435\u0433\u043E\u0432\u043E\u0440"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	679 u"\u044F\u0442\u043F\u043E\u0440\u0443\u0441\u0441\u043A"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	680 u"\u0438",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	681 "b1abfaaepdrnnbgefbaDotcwatmq2g4l"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	682
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	683 # (J) Spanish: Porqu<eacute>nopuedensimplementehablarenEspa<ntilde>ol
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	684 (u"\u0050\u006F\u0072\u0071\u0075\u00E9\u006E\u006F\u0070"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	685 u"\u0075\u0065\u0064\u0065\u006E\u0073\u0069\u006D\u0070"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	686 u"\u006C\u0065\u006D\u0065\u006E\u0074\u0065\u0068\u0061"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	687 u"\u0062\u006C\u0061\u0072\u0065\u006E\u0045\u0073\u0070"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	688 u"\u0061\u00F1\u006F\u006C",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	689 "PorqunopuedensimplementehablarenEspaol-fmd56a"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	690
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	691 # (K) Vietnamese:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	692 # T<adotbelow>isaoh<odotbelow>kh<ocirc>ngth<ecirchookabove>ch\
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	693 # <ihookabove>n<oacute>iti<ecircacute>ngVi<ecircdotbelow>t
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	694 (u"\u0054\u1EA1\u0069\u0073\u0061\u006F\u0068\u1ECD\u006B"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	695 u"\u0068\u00F4\u006E\u0067\u0074\u0068\u1EC3\u0063\u0068"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	696 u"\u1EC9\u006E\u00F3\u0069\u0074\u0069\u1EBF\u006E\u0067"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	697 u"\u0056\u0069\u1EC7\u0074",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	698 "TisaohkhngthchnitingVit-kjcr8268qyxafd2f1b9g"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	699
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	700 #(L) 3<nen>B<gumi><kinpachi><sensei>
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	701 (u"\u0033\u5E74\u0042\u7D44\u91D1\u516B\u5148\u751F",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	702 "3B-ww4c5e180e575a65lsy2b"),
28774 c9bee11eaa8b Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 28678 diff changeset	703
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	704 # (M) <amuro><namie>-with-SUPER-MONKEYS
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	705 (u"\u5B89\u5BA4\u5948\u7F8E\u6075\u002D\u0077\u0069\u0074"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	706 u"\u0068\u002D\u0053\u0055\u0050\u0045\u0052\u002D\u004D"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	707 u"\u004F\u004E\u004B\u0045\u0059\u0053",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	708 "-with-SUPER-MONKEYS-pc58ag80a8qai00g7n9n"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	709
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	710 # (N) Hello-Another-Way-<sorezore><no><basho>
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	711 (u"\u0048\u0065\u006C\u006C\u006F\u002D\u0041\u006E\u006F"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	712 u"\u0074\u0068\u0065\u0072\u002D\u0057\u0061\u0079\u002D"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	713 u"\u305D\u308C\u305E\u308C\u306E\u5834\u6240",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	714 "Hello-Another-Way--fc4qua05auwb3674vfr0b"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	715
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	716 # (O) <hitotsu><yane><no><shita>2
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	717 (u"\u3072\u3068\u3064\u5C4B\u6839\u306E\u4E0B\u0032",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	718 "2-u9tlzr9756bt3uc0v"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	719
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	720 # (P) Maji<de>Koi<suru>5<byou><mae>
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	721 (u"\u004D\u0061\u006A\u0069\u3067\u004B\u006F\u0069\u3059"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	722 u"\u308B\u0035\u79D2\u524D",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	723 "MajiKoi5-783gue6qz075azm5e"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	724
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	725 # (Q) <pafii>de<runba>
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	726 (u"\u30D1\u30D5\u30A3\u30FC\u0064\u0065\u30EB\u30F3\u30D0",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	727 "de-jg4avhby1noc0d"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	728
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	729 # (R) <sono><supiido><de>
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	730 (u"\u305D\u306E\u30B9\u30D4\u30FC\u30C9\u3067",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	731 "d9juau41awczczp"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	732
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	733 # (S) -> $1.00 <-
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	734 (u"\u002D\u003E\u0020\u0024\u0031\u002E\u0030\u0030\u0020"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	735 u"\u003C\u002D",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	736 "-> $1.00 <--")
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	737 ]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	738
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	739 for i in punycode_testcases:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	740 if len(i)!=2:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	741 print repr(i)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	742
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	743 class PunycodeTest(unittest.TestCase):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	744 def test_encode(self):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	745 for uni, puny in punycode_testcases:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	746 # Need to convert both strings to lower case, since
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	747 # some of the extended encodings use upper case, but our
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	748 # code produces only lower case. Converting just puny to
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	749 # lower is also insufficient, since some of the input characters
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	750 # are upper case.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	751 self.assertEquals(uni.encode("punycode").lower(), puny.lower())
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	752
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	753 def test_decode(self):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	754 for uni, puny in punycode_testcases:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	755 self.assertEquals(uni, puny.decode("punycode"))
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	756
35484 4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	757 class UnicodeInternalTest(unittest.TestCase):
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	758 def test_bug1251300(self):
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	759 # Decoding with unicode_internal used to not correctly handle "code
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	760 # points" above 0x10ffff on UCS-4 builds.
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	761 if sys.maxunicode > 0xffff:
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	762 ok = [
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	763 ("\x00\x10\xff\xff", u"\U0010ffff"),
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	764 ("\x00\x00\x01\x01", u"\U00000101"),
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	765 ("", u""),
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	766 ]
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	767 not_ok = [
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	768 "\x7f\xff\xff\xff",
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	769 "\x80\x00\x00\x00",
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	770 "\x81\x00\x00\x00",
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	771 "\x00",
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	772 "\x00\x00\x00\x00\x00",
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	773 ]
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	774 for internal, uni in ok:
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	775 if sys.byteorder == "little":
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	776 internal = "".join(reversed(internal))
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	777 self.assertEquals(uni, internal.decode("unicode_internal"))
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	778 for internal in not_ok:
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	779 if sys.byteorder == "little":
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	780 internal = "".join(reversed(internal))
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	781 self.assertRaises(UnicodeDecodeError, internal.decode,
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	782 "unicode_internal")
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	783
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	784 def test_decode_error_attributes(self):
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	785 if sys.maxunicode > 0xffff:
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	786 try:
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	787 "\x00\x00\x00\x00\x00\x11\x11\x00".decode("unicode_internal")
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	788 except UnicodeDecodeError, ex:
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	789 self.assertEquals("unicode_internal", ex.encoding)
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	790 self.assertEquals("\x00\x00\x00\x00\x00\x11\x11\x00", ex.object)
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	791 self.assertEquals(4, ex.start)
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	792 self.assertEquals(8, ex.end)
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	793 else:
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	794 self.fail()
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	795
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	796 def test_decode_callback(self):
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	797 if sys.maxunicode > 0xffff:
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	798 codecs.register_error("UnicodeInternalTest", codecs.ignore_errors)
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	799 decoder = codecs.getdecoder("unicode_internal")
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	800 ab = u"ab".encode("unicode_internal")
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	801 ignored = decoder("%s\x22\x22\x22\x22%s" % (ab[:4], ab[4:]),
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	802 "UnicodeInternalTest")
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	803 self.assertEquals((u"ab", 12), ignored)
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	804
53855 7dca250343ea Issue 3739: The unicode-internal encoder now reports the number of characters Walter Dörwald <walter@livinglogic.de> parents: 43932 diff changeset	805 def test_encode_length(self):
7dca250343ea Issue 3739: The unicode-internal encoder now reports the number of characters Walter Dörwald <walter@livinglogic.de> parents: 43932 diff changeset	806 # Issue 3739
7dca250343ea Issue 3739: The unicode-internal encoder now reports the number of characters Walter Dörwald <walter@livinglogic.de> parents: 43932 diff changeset	807 encoder = codecs.getencoder("unicode_internal")
7dca250343ea Issue 3739: The unicode-internal encoder now reports the number of characters Walter Dörwald <walter@livinglogic.de> parents: 43932 diff changeset	808 self.assertEquals(encoder(u"a")[1], 1)
7dca250343ea Issue 3739: The unicode-internal encoder now reports the number of characters Walter Dörwald <walter@livinglogic.de> parents: 43932 diff changeset	809 self.assertEquals(encoder(u"\xe9\u0142")[1], 2)
7dca250343ea Issue 3739: The unicode-internal encoder now reports the number of characters Walter Dörwald <walter@livinglogic.de> parents: 43932 diff changeset	810
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	811 # From http://www.gnu.org/software/libidn/draft-josefsson-idn-test-vectors.html
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	812 nameprep_tests = [
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	813 # 3.1 Map to nothing.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	814 ('foo\xc2\xad\xcd\x8f\xe1\xa0\x86\xe1\xa0\x8bbar'
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	815 '\xe2\x80\x8b\xe2\x81\xa0baz\xef\xb8\x80\xef\xb8\x88\xef'
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	816 '\xb8\x8f\xef\xbb\xbf',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	817 'foobarbaz'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	818 # 3.2 Case folding ASCII U+0043 U+0041 U+0046 U+0045.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	819 ('CAFE',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	820 'cafe'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	821 # 3.3 Case folding 8bit U+00DF (german sharp s).
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	822 # The original test case is bogus; it says \xc3\xdf
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	823 ('\xc3\x9f',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	824 'ss'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	825 # 3.4 Case folding U+0130 (turkish capital I with dot).
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	826 ('\xc4\xb0',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	827 'i\xcc\x87'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	828 # 3.5 Case folding multibyte U+0143 U+037A.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	829 ('\xc5\x83\xcd\xba',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	830 '\xc5\x84 \xce\xb9'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	831 # 3.6 Case folding U+2121 U+33C6 U+1D7BB.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	832 # XXX: skip this as it fails in UCS-2 mode
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	833 #('\xe2\x84\xa1\xe3\x8f\x86\xf0\x9d\x9e\xbb',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	834 # 'telc\xe2\x88\x95kg\xcf\x83'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	835 (None, None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	836 # 3.7 Normalization of U+006a U+030c U+00A0 U+00AA.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	837 ('j\xcc\x8c\xc2\xa0\xc2\xaa',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	838 '\xc7\xb0 a'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	839 # 3.8 Case folding U+1FB7 and normalization.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	840 ('\xe1\xbe\xb7',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	841 '\xe1\xbe\xb6\xce\xb9'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	842 # 3.9 Self-reverting case folding U+01F0 and normalization.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	843 # The original test case is bogus, it says `\xc7\xf0'
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	844 ('\xc7\xb0',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	845 '\xc7\xb0'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	846 # 3.10 Self-reverting case folding U+0390 and normalization.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	847 ('\xce\x90',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	848 '\xce\x90'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	849 # 3.11 Self-reverting case folding U+03B0 and normalization.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	850 ('\xce\xb0',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	851 '\xce\xb0'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	852 # 3.12 Self-reverting case folding U+1E96 and normalization.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	853 ('\xe1\xba\x96',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	854 '\xe1\xba\x96'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	855 # 3.13 Self-reverting case folding U+1F56 and normalization.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	856 ('\xe1\xbd\x96',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	857 '\xe1\xbd\x96'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	858 # 3.14 ASCII space character U+0020.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	859 (' ',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	860 ' '),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	861 # 3.15 Non-ASCII 8bit space character U+00A0.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	862 ('\xc2\xa0',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	863 ' '),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	864 # 3.16 Non-ASCII multibyte space character U+1680.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	865 ('\xe1\x9a\x80',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	866 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	867 # 3.17 Non-ASCII multibyte space character U+2000.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	868 ('\xe2\x80\x80',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	869 ' '),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	870 # 3.18 Zero Width Space U+200b.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	871 ('\xe2\x80\x8b',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	872 ''),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	873 # 3.19 Non-ASCII multibyte space character U+3000.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	874 ('\xe3\x80\x80',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	875 ' '),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	876 # 3.20 ASCII control characters U+0010 U+007F.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	877 ('\x10\x7f',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	878 '\x10\x7f'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	879 # 3.21 Non-ASCII 8bit control character U+0085.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	880 ('\xc2\x85',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	881 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	882 # 3.22 Non-ASCII multibyte control character U+180E.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	883 ('\xe1\xa0\x8e',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	884 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	885 # 3.23 Zero Width No-Break Space U+FEFF.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	886 ('\xef\xbb\xbf',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	887 ''),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	888 # 3.24 Non-ASCII control character U+1D175.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	889 ('\xf0\x9d\x85\xb5',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	890 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	891 # 3.25 Plane 0 private use character U+F123.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	892 ('\xef\x84\xa3',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	893 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	894 # 3.26 Plane 15 private use character U+F1234.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	895 ('\xf3\xb1\x88\xb4',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	896 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	897 # 3.27 Plane 16 private use character U+10F234.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	898 ('\xf4\x8f\x88\xb4',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	899 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	900 # 3.28 Non-character code point U+8FFFE.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	901 ('\xf2\x8f\xbf\xbe',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	902 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	903 # 3.29 Non-character code point U+10FFFF.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	904 ('\xf4\x8f\xbf\xbf',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	905 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	906 # 3.30 Surrogate code U+DF42.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	907 ('\xed\xbd\x82',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	908 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	909 # 3.31 Non-plain text character U+FFFD.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	910 ('\xef\xbf\xbd',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	911 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	912 # 3.32 Ideographic description character U+2FF5.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	913 ('\xe2\xbf\xb5',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	914 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	915 # 3.33 Display property character U+0341.
28774 c9bee11eaa8b Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 28678 diff changeset	916 ('\xcd\x81',
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	917 '\xcc\x81'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	918 # 3.34 Left-to-right mark U+200E.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	919 ('\xe2\x80\x8e',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	920 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	921 # 3.35 Deprecated U+202A.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	922 ('\xe2\x80\xaa',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	923 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	924 # 3.36 Language tagging character U+E0001.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	925 ('\xf3\xa0\x80\x81',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	926 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	927 # 3.37 Language tagging character U+E0042.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	928 ('\xf3\xa0\x81\x82',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	929 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	930 # 3.38 Bidi: RandALCat character U+05BE and LCat characters.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	931 ('foo\xd6\xbebar',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	932 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	933 # 3.39 Bidi: RandALCat character U+FD50 and LCat characters.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	934 ('foo\xef\xb5\x90bar',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	935 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	936 # 3.40 Bidi: RandALCat character U+FB38 and LCat characters.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	937 ('foo\xef\xb9\xb6bar',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	938 'foo \xd9\x8ebar'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	939 # 3.41 Bidi: RandALCat without trailing RandALCat U+0627 U+0031.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	940 ('\xd8\xa71',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	941 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	942 # 3.42 Bidi: RandALCat character U+0627 U+0031 U+0628.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	943 ('\xd8\xa71\xd8\xa8',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	944 '\xd8\xa71\xd8\xa8'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	945 # 3.43 Unassigned code point U+E0002.
28678 31cb8e0260db Skip nameprep test 3.43, as we do allow unassigned characters. The test Martin v. Löwis <martin@v.loewis.de> parents: 28671 diff changeset	946 # Skip this test as we allow unassigned
31cb8e0260db Skip nameprep test 3.43, as we do allow unassigned characters. The test Martin v. Löwis <martin@v.loewis.de> parents: 28671 diff changeset	947 #('\xf3\xa0\x80\x82',
31cb8e0260db Skip nameprep test 3.43, as we do allow unassigned characters. The test Martin v. Löwis <martin@v.loewis.de> parents: 28671 diff changeset	948 # None),
31cb8e0260db Skip nameprep test 3.43, as we do allow unassigned characters. The test Martin v. Löwis <martin@v.loewis.de> parents: 28671 diff changeset	949 (None, None),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	950 # 3.44 Larger test (shrinking).
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	951 # Original test case reads \xc3\xdf
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	952 ('X\xc2\xad\xc3\x9f\xc4\xb0\xe2\x84\xa1j\xcc\x8c\xc2\xa0\xc2'
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	953 '\xaa\xce\xb0\xe2\x80\x80',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	954 'xssi\xcc\x87tel\xc7\xb0 a\xce\xb0 '),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	955 # 3.45 Larger test (expanding).
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	956 # Original test case reads \xc3\x9f
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	957 ('X\xc3\x9f\xe3\x8c\x96\xc4\xb0\xe2\x84\xa1\xe2\x92\x9f\xe3\x8c'
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	958 '\x80',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	959 'xss\xe3\x82\xad\xe3\x83\xad\xe3\x83\xa1\xe3\x83\xbc\xe3'
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	960 '\x83\x88\xe3\x83\xabi\xcc\x87tel\x28d\x29\xe3\x82'
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	961 '\xa2\xe3\x83\x91\xe3\x83\xbc\xe3\x83\x88')
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	962 ]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	963
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	964
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	965 class NameprepTest(unittest.TestCase):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	966 def test_nameprep(self):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	967 from encodings.idna import nameprep
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	968 for pos, (orig, prepped) in enumerate(nameprep_tests):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	969 if orig is None:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	970 # Skipped
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	971 continue
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	972 # The Unicode strings are given in UTF-8
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	973 orig = unicode(orig, "utf-8")
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	974 if prepped is None:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	975 # Input contains prohibited characters
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	976 self.assertRaises(UnicodeError, nameprep, orig)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	977 else:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	978 prepped = unicode(prepped, "utf-8")
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	979 try:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	980 self.assertEquals(nameprep(orig), prepped)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	981 except Exception,e:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	982 raise test_support.TestFailed("Test 3.%d: %s" % (pos+1, str(e)))
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	983
37719 cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	984 class IDNACodecTest(unittest.TestCase):
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	985 def test_builtin_decode(self):
31795 bf37429f6d9f Add test case for unicode(somestring, "idna"). Martin v. Löwis <martin@v.loewis.de> parents: 28917 diff changeset	986 self.assertEquals(unicode("python.org", "idna"), u"python.org")
37719 cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	987 self.assertEquals(unicode("python.org.", "idna"), u"python.org.")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	988 self.assertEquals(unicode("xn--pythn-mua.org", "idna"), u"pyth\xf6n.org")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	989 self.assertEquals(unicode("xn--pythn-mua.org.", "idna"), u"pyth\xf6n.org.")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	990
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	991 def test_builtin_encode(self):
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	992 self.assertEquals(u"python.org".encode("idna"), "python.org")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	993 self.assertEquals("python.org.".encode("idna"), "python.org.")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	994 self.assertEquals(u"pyth\xf6n.org".encode("idna"), "xn--pythn-mua.org")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	995 self.assertEquals(u"pyth\xf6n.org.".encode("idna"), "xn--pythn-mua.org.")
31795 bf37429f6d9f Add test case for unicode(somestring, "idna"). Martin v. Löwis <martin@v.loewis.de> parents: 28917 diff changeset	996
35435 9d343bc81863 Make IDNA return an empty string when the input is empty. Fixes #1163178. Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	997 def test_stream(self):
9d343bc81863 Make IDNA return an empty string when the input is empty. Fixes #1163178. Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	998 import StringIO
9d343bc81863 Make IDNA return an empty string when the input is empty. Fixes #1163178. Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	999 r = codecs.getreader("idna")(StringIO.StringIO("abc"))
9d343bc81863 Make IDNA return an empty string when the input is empty. Fixes #1163178. Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	1000 r.read(3)
9d343bc81863 Make IDNA return an empty string when the input is empty. Fixes #1163178. Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	1001 self.assertEquals(r.read(), u"")
9d343bc81863 Make IDNA return an empty string when the input is empty. Fixes #1163178. Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	1002
37719 cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1003 def test_incremental_decode(self):
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1004 self.assertEquals(
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1005 "".join(codecs.iterdecode("python.org", "idna")),
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1006 u"python.org"
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1007 )
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1008 self.assertEquals(
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1009 "".join(codecs.iterdecode("python.org.", "idna")),
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1010 u"python.org."
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1011 )
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1012 self.assertEquals(
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1013 "".join(codecs.iterdecode("xn--pythn-mua.org.", "idna")),
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1014 u"pyth\xf6n.org."
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1015 )
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1016 self.assertEquals(
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1017 "".join(codecs.iterdecode("xn--pythn-mua.org.", "idna")),
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1018 u"pyth\xf6n.org."
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1019 )
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1020
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1021 decoder = codecs.getincrementaldecoder("idna")()
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1022 self.assertEquals(decoder.decode("xn--xam", ), u"")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1023 self.assertEquals(decoder.decode("ple-9ta.o", ), u"\xe4xample.")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1024 self.assertEquals(decoder.decode(u"rg"), u"")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1025 self.assertEquals(decoder.decode(u"", True), u"org")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1026
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1027 decoder.reset()
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1028 self.assertEquals(decoder.decode("xn--xam", ), u"")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1029 self.assertEquals(decoder.decode("ple-9ta.o", ), u"\xe4xample.")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1030 self.assertEquals(decoder.decode("rg."), u"org.")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1031 self.assertEquals(decoder.decode("", True), u"")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1032
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1033 def test_incremental_encode(self):
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1034 self.assertEquals(
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1035 "".join(codecs.iterencode(u"python.org", "idna")),
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1036 "python.org"
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1037 )
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1038 self.assertEquals(
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1039 "".join(codecs.iterencode(u"python.org.", "idna")),
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1040 "python.org."
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1041 )
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1042 self.assertEquals(
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1043 "".join(codecs.iterencode(u"pyth\xf6n.org.", "idna")),
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1044 "xn--pythn-mua.org."
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1045 )
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1046 self.assertEquals(
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1047 "".join(codecs.iterencode(u"pyth\xf6n.org.", "idna")),
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1048 "xn--pythn-mua.org."
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1049 )
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1050
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1051 encoder = codecs.getincrementalencoder("idna")()
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1052 self.assertEquals(encoder.encode(u"\xe4x"), "")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1053 self.assertEquals(encoder.encode(u"ample.org"), "xn--xample-9ta.")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1054 self.assertEquals(encoder.encode(u"", True), "org")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1055
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1056 encoder.reset()
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1057 self.assertEquals(encoder.encode(u"\xe4x"), "")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1058 self.assertEquals(encoder.encode(u"ample.org."), "xn--xample-9ta.org.")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1059 self.assertEquals(encoder.encode(u"", True), "")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1060
32572 8ea2cb1ec598 Add generic codecs.encode() and .decode() APIs that don't impose Marc-André Lemburg <mal@egenix.com> parents: 32538 diff changeset	1061 class CodecsModuleTest(unittest.TestCase):
8ea2cb1ec598 Add generic codecs.encode() and .decode() APIs that don't impose Marc-André Lemburg <mal@egenix.com> parents: 32538 diff changeset	1062
8ea2cb1ec598 Add generic codecs.encode() and .decode() APIs that don't impose Marc-André Lemburg <mal@egenix.com> parents: 32538 diff changeset	1063 def test_decode(self):
8ea2cb1ec598 Add generic codecs.encode() and .decode() APIs that don't impose Marc-André Lemburg <mal@egenix.com> parents: 32538 diff changeset	1064 self.assertEquals(codecs.decode('\xe4\xf6\xfc', 'latin-1'),
8ea2cb1ec598 Add generic codecs.encode() and .decode() APIs that don't impose Marc-André Lemburg <mal@egenix.com> parents: 32538 diff changeset	1065 u'\xe4\xf6\xfc')
33855 1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1066 self.assertRaises(TypeError, codecs.decode)
1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1067 self.assertEquals(codecs.decode('abc'), u'abc')
1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1068 self.assertRaises(UnicodeDecodeError, codecs.decode, '\xff', 'ascii')
1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1069
32572 8ea2cb1ec598 Add generic codecs.encode() and .decode() APIs that don't impose Marc-André Lemburg <mal@egenix.com> parents: 32538 diff changeset	1070 def test_encode(self):
8ea2cb1ec598 Add generic codecs.encode() and .decode() APIs that don't impose Marc-André Lemburg <mal@egenix.com> parents: 32538 diff changeset	1071 self.assertEquals(codecs.encode(u'\xe4\xf6\xfc', 'latin-1'),
8ea2cb1ec598 Add generic codecs.encode() and .decode() APIs that don't impose Marc-André Lemburg <mal@egenix.com> parents: 32538 diff changeset	1072 '\xe4\xf6\xfc')
33855 1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1073 self.assertRaises(TypeError, codecs.encode)
35889 e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1074 self.assertRaises(LookupError, codecs.encode, "foo", "__spam__")
33855 1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1075 self.assertEquals(codecs.encode(u'abc'), 'abc')
1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1076 self.assertRaises(UnicodeEncodeError, codecs.encode, u'\xffff', 'ascii')
1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1077
1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1078 def test_register(self):
1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1079 self.assertRaises(TypeError, codecs.register)
35889 e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1080 self.assertRaises(TypeError, codecs.register, 42)
33855 1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1081
1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1082 def test_lookup(self):
1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1083 self.assertRaises(TypeError, codecs.lookup)
1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1084 self.assertRaises(LookupError, codecs.lookup, "__spam__")
35889 e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1085 self.assertRaises(LookupError, codecs.lookup, " ")
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1086
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1087 def test_getencoder(self):
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1088 self.assertRaises(TypeError, codecs.getencoder)
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1089 self.assertRaises(LookupError, codecs.getencoder, "__spam__")
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1090
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1091 def test_getdecoder(self):
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1092 self.assertRaises(TypeError, codecs.getdecoder)
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1093 self.assertRaises(LookupError, codecs.getdecoder, "__spam__")
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1094
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1095 def test_getreader(self):
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1096 self.assertRaises(TypeError, codecs.getreader)
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1097 self.assertRaises(LookupError, codecs.getreader, "__spam__")
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1098
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1099 def test_getwriter(self):
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1100 self.assertRaises(TypeError, codecs.getwriter)
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1101 self.assertRaises(LookupError, codecs.getwriter, "__spam__")
32572 8ea2cb1ec598 Add generic codecs.encode() and .decode() APIs that don't impose Marc-André Lemburg <mal@egenix.com> parents: 32538 diff changeset	1102
33782 a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1103 class StreamReaderTest(unittest.TestCase):
a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1104
a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1105 def setUp(self):
a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1106 self.reader = codecs.getreader('utf-8')
a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1107 self.stream = StringIO.StringIO('\xed\x95\x9c\n\xea\xb8\x80')
a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1108
a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1109 def test_readlines(self):
a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1110 f = self.reader(self.stream)
a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1111 self.assertEquals(f.readlines(), [u'\ud55c\n', u'\uae00'])
a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1112
40239 7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1113 class EncodedFileTest(unittest.TestCase):
40284 ff678669aa6e Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 40274 diff changeset	1114
40239 7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1115 def test_basic(self):
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1116 f = StringIO.StringIO('\xed\x95\x9c\n\xea\xb8\x80')
40249 9663100c95ca Fix the new EncodedFile test to work with big endian platforms. Georg Brandl <georg@python.org> parents: 40239 diff changeset	1117 ef = codecs.EncodedFile(f, 'utf-16-le', 'utf-8')
9663100c95ca Fix the new EncodedFile test to work with big endian platforms. Georg Brandl <georg@python.org> parents: 40239 diff changeset	1118 self.assertEquals(ef.read(), '\\\xd5\n\x00\x00\xae')
40239 7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1119
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1120 f = StringIO.StringIO()
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1121 ef = codecs.EncodedFile(f, 'utf-8', 'latin1')
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1122 ef.write('\xc3\xbc')
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1123 self.assertEquals(f.getvalue(), '\xfc')
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1124
35269 2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1125 class Str2StrTest(unittest.TestCase):
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1126
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1127 def test_read(self):
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1128 sin = "\x80".encode("base64_codec")
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1129 reader = codecs.getreader("base64_codec")(StringIO.StringIO(sin))
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1130 sout = reader.read()
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1131 self.assertEqual(sout, "\x80")
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1132 self.assert_(isinstance(sout, str))
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1133
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1134 def test_readline(self):
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1135 sin = "\x80".encode("base64_codec")
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1136 reader = codecs.getreader("base64_codec")(StringIO.StringIO(sin))
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1137 sout = reader.readline()
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1138 self.assertEqual(sout, "\x80")
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1139 self.assert_(isinstance(sout, str))
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1140
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1141 all_unicode_encodings = [
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1142 "ascii",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1143 "base64_codec",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1144 "big5",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1145 "big5hkscs",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1146 "charmap",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1147 "cp037",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1148 "cp1006",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1149 "cp1026",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1150 "cp1140",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1151 "cp1250",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1152 "cp1251",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1153 "cp1252",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1154 "cp1253",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1155 "cp1254",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1156 "cp1255",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1157 "cp1256",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1158 "cp1257",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1159 "cp1258",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1160 "cp424",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1161 "cp437",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1162 "cp500",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1163 "cp737",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1164 "cp775",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1165 "cp850",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1166 "cp852",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1167 "cp855",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1168 "cp856",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1169 "cp857",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1170 "cp860",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1171 "cp861",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1172 "cp862",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1173 "cp863",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1174 "cp864",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1175 "cp865",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1176 "cp866",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1177 "cp869",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1178 "cp874",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1179 "cp875",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1180 "cp932",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1181 "cp949",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1182 "cp950",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1183 "euc_jis_2004",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1184 "euc_jisx0213",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1185 "euc_jp",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1186 "euc_kr",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1187 "gb18030",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1188 "gb2312",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1189 "gbk",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1190 "hex_codec",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1191 "hp_roman8",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1192 "hz",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1193 "idna",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1194 "iso2022_jp",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1195 "iso2022_jp_1",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1196 "iso2022_jp_2",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1197 "iso2022_jp_2004",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1198 "iso2022_jp_3",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1199 "iso2022_jp_ext",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1200 "iso2022_kr",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1201 "iso8859_1",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1202 "iso8859_10",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1203 "iso8859_11",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1204 "iso8859_13",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1205 "iso8859_14",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1206 "iso8859_15",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1207 "iso8859_16",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1208 "iso8859_2",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1209 "iso8859_3",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1210 "iso8859_4",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1211 "iso8859_5",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1212 "iso8859_6",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1213 "iso8859_7",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1214 "iso8859_8",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1215 "iso8859_9",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1216 "johab",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1217 "koi8_r",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1218 "koi8_u",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1219 "latin_1",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1220 "mac_cyrillic",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1221 "mac_greek",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1222 "mac_iceland",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1223 "mac_latin2",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1224 "mac_roman",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1225 "mac_turkish",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1226 "palmos",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1227 "ptcp154",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1228 "punycode",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1229 "raw_unicode_escape",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1230 "rot_13",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1231 "shift_jis",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1232 "shift_jis_2004",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1233 "shift_jisx0213",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1234 "tis_620",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1235 "unicode_escape",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1236 "unicode_internal",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1237 "utf_16",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1238 "utf_16_be",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1239 "utf_16_le",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1240 "utf_7",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1241 "utf_8",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1242 ]
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1243
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1244 if hasattr(codecs, "mbcs_encode"):
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1245 all_unicode_encodings.append("mbcs")
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1246
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1247 # The following encodings work only with str, not unicode
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1248 all_string_encodings = [
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1249 "quopri_codec",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1250 "string_escape",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1251 "uu_codec",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1252 ]
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1253
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1254 # The following encoding is not tested, because it's not supposed
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1255 # to work:
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1256 # "undefined"
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1257
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1258 # The following encodings don't work in stateful mode
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1259 broken_unicode_with_streams = [
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1260 "base64_codec",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1261 "hex_codec",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1262 "punycode",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1263 "unicode_internal"
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1264 ]
40251 e16800ac1eea Bug #1586613: fix zlib and bz2 codecs' incremental en/decoders. Georg Brandl <georg@python.org> parents: 40249 diff changeset	1265 broken_incremental_coders = broken_unicode_with_streams[:]
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1266
40271 c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1267 # The following encodings only support "strict" mode
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1268 only_strict_mode = [
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1269 "idna",
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1270 "zlib_codec",
40274 bc8ffc3e3cdb I'm assuming this is correct, it fixes the tests so they pass again Neal Norwitz <nnorwitz@gmail.com> parents: 40271 diff changeset	1271 "bz2_codec",
40271 c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1272 ]
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1273
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1274 try:
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1275 import bz2
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1276 except ImportError:
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1277 pass
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1278 else:
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1279 all_unicode_encodings.append("bz2_codec")
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1280 broken_unicode_with_streams.append("bz2_codec")
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1281
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1282 try:
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1283 import zlib
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1284 except ImportError:
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1285 pass
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1286 else:
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1287 all_unicode_encodings.append("zlib_codec")
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1288 broken_unicode_with_streams.append("zlib_codec")
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1289
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1290 class BasicUnicodeTest(unittest.TestCase):
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1291 def test_basics(self):
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1292 s = u"abc123" # all codecs should be able to encode these
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1293 for encoding in all_unicode_encodings:
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1294 name = codecs.lookup(encoding).name
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1295 if encoding.endswith("_codec"):
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1296 name += "_codec"
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1297 elif encoding == "latin_1":
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1298 name = "latin_1"
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1299 self.assertEqual(encoding.replace("_", "-"), name.replace("_", "-"))
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1300 (bytes, size) = codecs.getencoder(encoding)(s)
53855 7dca250343ea Issue 3739: The unicode-internal encoder now reports the number of characters Walter Dörwald <walter@livinglogic.de> parents: 43932 diff changeset	1301 self.assertEqual(size, len(s), "%r != %r (encoding=%r)" % (size, len(s), encoding))
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1302 (chars, size) = codecs.getdecoder(encoding)(bytes)
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1303 self.assertEqual(chars, s, "%r != %r (encoding=%r)" % (chars, s, encoding))
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1304
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1305 if encoding not in broken_unicode_with_streams:
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1306 # check stream reader/writer
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1307 q = Queue()
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1308 writer = codecs.getwriter(encoding)(q)
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1309 encodedresult = ""
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1310 for c in s:
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1311 writer.write(c)
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1312 encodedresult += q.read()
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1313 q = Queue()
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1314 reader = codecs.getreader(encoding)(q)
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1315 decodedresult = u""
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1316 for c in encodedresult:
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1317 q.write(c)
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1318 decodedresult += reader.read()
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1319 self.assertEqual(decodedresult, s, "%r != %r (encoding=%r)" % (decodedresult, s, encoding))
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1320
40251 e16800ac1eea Bug #1586613: fix zlib and bz2 codecs' incremental en/decoders. Georg Brandl <georg@python.org> parents: 40249 diff changeset	1321 if encoding not in broken_incremental_coders:
37077 38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	1322 # check incremental decoder/encoder (fetched via the Python
38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	1323 # and C API) and iterencode()/iterdecode()
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1324 try:
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1325 encoder = codecs.getincrementalencoder(encoding)()
37077 38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	1326 cencoder = _testcapi.codec_incrementalencoder(encoding)
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1327 except LookupError: # no IncrementalEncoder
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1328 pass
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1329 else:
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1330 # check incremental decoder/encoder
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1331 encodedresult = ""
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1332 for c in s:
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1333 encodedresult += encoder.encode(c)
37708 230fff6e0618 Call encode()/decode() with final==True as the last call in the Walter Dörwald <walter@livinglogic.de> parents: 37077 diff changeset	1334 encodedresult += encoder.encode(u"", True)
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1335 decoder = codecs.getincrementaldecoder(encoding)()
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1336 decodedresult = u""
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1337 for c in encodedresult:
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1338 decodedresult += decoder.decode(c)
37708 230fff6e0618 Call encode()/decode() with final==True as the last call in the Walter Dörwald <walter@livinglogic.de> parents: 37077 diff changeset	1339 decodedresult += decoder.decode("", True)
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1340 self.assertEqual(decodedresult, s, "%r != %r (encoding=%r)" % (decodedresult, s, encoding))
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1341
37077 38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	1342 # check C API
38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	1343 encodedresult = ""
38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	1344 for c in s:
38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	1345 encodedresult += cencoder.encode(c)
37708 230fff6e0618 Call encode()/decode() with final==True as the last call in the Walter Dörwald <walter@livinglogic.de> parents: 37077 diff changeset	1346 encodedresult += cencoder.encode(u"", True)
37077 38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	1347 cdecoder = _testcapi.codec_incrementaldecoder(encoding)
38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	1348 decodedresult = u""
38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	1349 for c in encodedresult:
38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	1350 decodedresult += cdecoder.decode(c)
37708 230fff6e0618 Call encode()/decode() with final==True as the last call in the Walter Dörwald <walter@livinglogic.de> parents: 37077 diff changeset	1351 decodedresult += cdecoder.decode("", True)
37077 38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	1352 self.assertEqual(decodedresult, s, "%r != %r (encoding=%r)" % (decodedresult, s, encoding))
38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	1353
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1354 # check iterencode()/iterdecode()
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1355 result = u"".join(codecs.iterdecode(codecs.iterencode(s, encoding), encoding))
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1356 self.assertEqual(result, s, "%r != %r (encoding=%r)" % (result, s, encoding))
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1357
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1358 # check iterencode()/iterdecode() with empty string
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1359 result = u"".join(codecs.iterdecode(codecs.iterencode(u"", encoding), encoding))
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1360 self.assertEqual(result, u"")
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1361
40271 c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1362 if encoding not in only_strict_mode:
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1363 # check incremental decoder/encoder with errors argument
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1364 try:
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1365 encoder = codecs.getincrementalencoder(encoding)("ignore")
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1366 cencoder = _testcapi.codec_incrementalencoder(encoding, "ignore")
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1367 except LookupError: # no IncrementalEncoder
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1368 pass
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1369 else:
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1370 encodedresult = "".join(encoder.encode(c) for c in s)
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1371 decoder = codecs.getincrementaldecoder(encoding)("ignore")
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1372 decodedresult = u"".join(decoder.decode(c) for c in encodedresult)
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1373 self.assertEqual(decodedresult, s, "%r != %r (encoding=%r)" % (decodedresult, s, encoding))
40284 ff678669aa6e Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 40274 diff changeset	1374
40271 c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1375 encodedresult = "".join(cencoder.encode(c) for c in s)
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1376 cdecoder = _testcapi.codec_incrementaldecoder(encoding, "ignore")
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1377 decodedresult = u"".join(cdecoder.decode(c) for c in encodedresult)
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1378 self.assertEqual(decodedresult, s, "%r != %r (encoding=%r)" % (decodedresult, s, encoding))
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1379
34750 679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1380 def test_seek(self):
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1381 # all codecs should be able to encode these
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1382 s = u"%s\n%s\n" % (100u"abc123", 100u"def456")
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1383 for encoding in all_unicode_encodings:
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1384 if encoding == "idna": # FIXME: See SF bug #1163178
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1385 continue
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1386 if encoding in broken_unicode_with_streams:
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1387 continue
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1388 reader = codecs.getreader(encoding)(StringIO.StringIO(s.encode(encoding)))
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1389 for t in xrange(5):
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1390 # Test that calling seek resets the internal codec state and buffers
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1391 reader.seek(0, 0)
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1392 line = reader.readline()
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1393 self.assertEqual(s[:len(line)], line)
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1394
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1395 def test_bad_decode_args(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1396 for encoding in all_unicode_encodings:
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1397 decoder = codecs.getdecoder(encoding)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1398 self.assertRaises(TypeError, decoder)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1399 if encoding not in ("idna", "punycode"):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1400 self.assertRaises(TypeError, decoder, 42)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1401
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1402 def test_bad_encode_args(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1403 for encoding in all_unicode_encodings:
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1404 encoder = codecs.getencoder(encoding)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1405 self.assertRaises(TypeError, encoder)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1406
38787 70b166ca1c49 Verify the crash due to EncodingMap not initialized does not return Neal Norwitz <nnorwitz@gmail.com> parents: 37719 diff changeset	1407 def test_encoding_map_type_initialized(self):
70b166ca1c49 Verify the crash due to EncodingMap not initialized does not return Neal Norwitz <nnorwitz@gmail.com> parents: 37719 diff changeset	1408 from encodings import cp1140
70b166ca1c49 Verify the crash due to EncodingMap not initialized does not return Neal Norwitz <nnorwitz@gmail.com> parents: 37719 diff changeset	1409 # This used to crash, we are only verifying there's no crash.
70b166ca1c49 Verify the crash due to EncodingMap not initialized does not return Neal Norwitz <nnorwitz@gmail.com> parents: 37719 diff changeset	1410 table_type = type(cp1140.encoding_table)
70b166ca1c49 Verify the crash due to EncodingMap not initialized does not return Neal Norwitz <nnorwitz@gmail.com> parents: 37719 diff changeset	1411 self.assertEqual(table_type, table_type)
70b166ca1c49 Verify the crash due to EncodingMap not initialized does not return Neal Norwitz <nnorwitz@gmail.com> parents: 37719 diff changeset	1412
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1413 class BasicStrTest(unittest.TestCase):
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1414 def test_basics(self):
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1415 s = "abc123"
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1416 for encoding in all_string_encodings:
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1417 (bytes, size) = codecs.getencoder(encoding)(s)
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1418 self.assertEqual(size, len(s))
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1419 (chars, size) = codecs.getdecoder(encoding)(bytes)
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1420 self.assertEqual(chars, s, "%r != %r (encoding=%r)" % (chars, s, encoding))
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1421
35702 22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1422 class CharmapTest(unittest.TestCase):
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1423 def test_decode_with_string_map(self):
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1424 self.assertEquals(
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1425 codecs.charmap_decode("\x00\x01\x02", "strict", u"abc"),
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1426 (u"abc", 3)
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1427 )
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1428
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1429 self.assertEquals(
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1430 codecs.charmap_decode("\x00\x01\x02", "replace", u"ab"),
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1431 (u"ab\ufffd", 3)
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1432 )
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1433
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1434 self.assertEquals(
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1435 codecs.charmap_decode("\x00\x01\x02", "replace", u"ab\ufffe"),
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1436 (u"ab\ufffd", 3)
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1437 )
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1438
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1439 self.assertEquals(
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1440 codecs.charmap_decode("\x00\x01\x02", "ignore", u"ab"),
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1441 (u"ab", 3)
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1442 )
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1443
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1444 self.assertEquals(
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1445 codecs.charmap_decode("\x00\x01\x02", "ignore", u"ab\ufffe"),
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1446 (u"ab", 3)
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1447 )
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1448
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1449 allbytes = "".join(chr(i) for i in xrange(256))
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1450 self.assertEquals(
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1451 codecs.charmap_decode(allbytes, "ignore", u""),
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1452 (u"", len(allbytes))
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1453 )
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1454
40239 7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1455 class WithStmtTest(unittest.TestCase):
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1456 def test_encodedfile(self):
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1457 f = StringIO.StringIO("\xc3\xbc")
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1458 with codecs.EncodedFile(f, "latin-1", "utf-8") as ef:
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1459 self.assertEquals(ef.read(), "\xfc")
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1460
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1461 def test_streamreaderwriter(self):
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1462 f = StringIO.StringIO("\xc3\xbc")
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1463 info = codecs.lookup("utf-8")
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1464 with codecs.StreamReaderWriter(f, info.streamreader,
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1465 info.streamwriter, 'strict') as srw:
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1466 self.assertEquals(srw.read(), u"\xfc")
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1467
35702 22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1468
20013 e3184e46303b Change the PyUnit-based tests to use the test_main() approach. This Fred Drake <fdrake@acm.org> parents: 18163 diff changeset	1469 def test_main():
28917 1741af48fd9f Combine the functionality of test_support.run_unittest() Walter Dörwald <walter@livinglogic.de> parents: 28774 diff changeset	1470 test_support.run_unittest(
42649 c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	1471 UTF32Test,
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	1472 UTF32LETest,
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	1473 UTF32BETest,
28917 1741af48fd9f Combine the functionality of test_support.run_unittest() Walter Dörwald <walter@livinglogic.de> parents: 28774 diff changeset	1474 UTF16Test,
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	1475 UTF16LETest,
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	1476 UTF16BETest,
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	1477 UTF8Test,
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	1478 UTF8SigTest,
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1479 UTF7Test,
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1480 UTF16ExTest,
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1481 ReadBufferTest,
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1482 CharBufferTest,
28917 1741af48fd9f Combine the functionality of test_support.run_unittest() Walter Dörwald <walter@livinglogic.de> parents: 28774 diff changeset	1483 EscapeDecodeTest,
1741af48fd9f Combine the functionality of test_support.run_unittest() Walter Dörwald <walter@livinglogic.de> parents: 28774 diff changeset	1484 RecodingTest,
1741af48fd9f Combine the functionality of test_support.run_unittest() Walter Dörwald <walter@livinglogic.de> parents: 28774 diff changeset	1485 PunycodeTest,
35484 4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	1486 UnicodeInternalTest,
31795 bf37429f6d9f Add test case for unicode(somestring, "idna"). Martin v. Löwis <martin@v.loewis.de> parents: 28917 diff changeset	1487 NameprepTest,
37719 cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1488 IDNACodecTest,
33782 a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1489 CodecsModuleTest,
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1490 StreamReaderTest,
40239 7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1491 EncodedFileTest,
35269 2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1492 Str2StrTest,
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1493 BasicUnicodeTest,
35702 22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1494 BasicStrTest,
40239 7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1495 CharmapTest,
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1496 WithStmtTest,
28917 1741af48fd9f Combine the functionality of test_support.run_unittest() Walter Dörwald <walter@livinglogic.de> parents: 28774 diff changeset	1497 )
20013 e3184e46303b Change the PyUnit-based tests to use the test_main() approach. This Fred Drake <fdrake@acm.org> parents: 18163 diff changeset	1498
e3184e46303b Change the PyUnit-based tests to use the test_main() approach. This Fred Drake <fdrake@acm.org> parents: 18163 diff changeset	1499
e3184e46303b Change the PyUnit-based tests to use the test_main() approach. This Fred Drake <fdrake@acm.org> parents: 18163 diff changeset	1500 if __name__ == "__main__":
e3184e46303b Change the PyUnit-based tests to use the test_main() approach. This Fred Drake <fdrake@acm.org> parents: 18163 diff changeset	1501 test_main()

Mercurial > cpython-lru_cache-weakref

annotate Lib/test/test_codecs.py @ 54294:dba767bc3155 legacy-trunk