cpython-lru_cache-weakref: Lib/test/test

annotate Lib/test/test_codecs.py @ 65035:87f4b32905df 2.7

Cosmetic fix to use print function.

author	Hirokazu Yamamoto <ocean-city@m2.ccsnet.ne.jp>
date	Fri, 01 Oct 2010 10:48:47 +0000
parents	98245464b665
children	614c84670c63

rev	line source
24364 aa0d2d51fbde Get rid of relative imports in all unittests. Now anything that Barry Warsaw <barry@python.org> parents: 20013 diff changeset	1 from test import test_support
aa0d2d51fbde Get rid of relative imports in all unittests. Now anything that Barry Warsaw <barry@python.org> parents: 20013 diff changeset	2 import unittest
18163 f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	3 import codecs
37077 38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	4 import sys, StringIO, _testcapi
18163 f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	5
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	6 class Queue(object):
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	7 """
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	8 queue: write bytes at one end, read bytes from the other end
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	9 """
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	10 def __init__(self):
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	11 self._buffer = ""
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	12
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	13 def write(self, chars):
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	14 self._buffer += chars
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	15
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	16 def read(self, size=-1):
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	17 if size<0:
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	18 s = self._buffer
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	19 self._buffer = ""
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	20 return s
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	21 else:
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	22 s = self._buffer[:size]
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	23 self._buffer = self._buffer[size:]
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	24 return s
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	25
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	26 class ReadTest(unittest.TestCase):
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	27 def check_partial(self, input, partialresults):
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	28 # get a StreamReader for the encoding and feed the bytestring version
43903 6aa8e135d9f8 Fix typo in comment. Walter Dörwald <walter@livinglogic.de> parents: 42649 diff changeset	29 # of input to the reader byte by byte. Read everything available from
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	30 # the StreamReader and check that the results equal the appropriate
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	31 # entries from partialresults.
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	32 q = Queue()
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	33 r = codecs.getreader(self.encoding)(q)
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	34 result = u""
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	35 for (c, partialresult) in zip(input.encode(self.encoding), partialresults):
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	36 q.write(c)
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	37 result += r.read()
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	38 self.assertEqual(result, partialresult)
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	39 # check that there's nothing left in the buffers
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	40 self.assertEqual(r.read(), u"")
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	41 self.assertEqual(r.bytebuffer, "")
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	42 self.assertEqual(r.charbuffer, u"")
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	43
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	44 # do the check again, this time using a incremental decoder
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	45 d = codecs.getincrementaldecoder(self.encoding)()
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	46 result = u""
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	47 for (c, partialresult) in zip(input.encode(self.encoding), partialresults):
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	48 result += d.decode(c)
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	49 self.assertEqual(result, partialresult)
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	50 # check that there's nothing left in the buffers
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	51 self.assertEqual(d.decode("", True), u"")
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	52 self.assertEqual(d.buffer, "")
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	53
43932 e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls. Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 43905 diff changeset	54 # Check whether the reset method works properly
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	55 d.reset()
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	56 result = u""
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	57 for (c, partialresult) in zip(input.encode(self.encoding), partialresults):
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	58 result += d.decode(c)
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	59 self.assertEqual(result, partialresult)
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	60 # check that there's nothing left in the buffers
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	61 self.assertEqual(d.decode("", True), u"")
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	62 self.assertEqual(d.buffer, "")
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	63
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	64 # check iterdecode()
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	65 encoded = input.encode(self.encoding)
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	66 self.assertEqual(
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	67 input,
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	68 u"".join(codecs.iterdecode(encoded, self.encoding))
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	69 )
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	70
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	71 def test_readline(self):
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	72 def getreader(input):
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	73 stream = StringIO.StringIO(input.encode(self.encoding))
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	74 return codecs.getreader(self.encoding)(stream)
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	75
36855 204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	76 def readalllines(input, keepends=True, size=None):
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	77 reader = getreader(input)
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	78 lines = []
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	79 while True:
36855 204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	80 line = reader.readline(size=size, keepends=keepends)
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	81 if not line:
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	82 break
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	83 lines.append(line)
36855 204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	84 return "\|".join(lines)
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	85
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	86 s = u"foo\nbar\r\nbaz\rspam\u2028eggs"
36855 204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	87 sexpected = u"foo\n\|bar\r\n\|baz\r\|spam\u2028\|eggs"
204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	88 sexpectednoends = u"foo\|bar\|baz\|spam\|eggs"
204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	89 self.assertEqual(readalllines(s, True), sexpected)
204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	90 self.assertEqual(readalllines(s, False), sexpectednoends)
204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	91 self.assertEqual(readalllines(s, True, 10), sexpected)
204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	92 self.assertEqual(readalllines(s, False, 10), sexpectednoends)
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	93
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	94 # Test long lines (multiple calls to read() in readline())
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	95 vw = []
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	96 vwo = []
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	97 for (i, lineend) in enumerate(u"\n \r\n \r \u2028".split()):
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	98 vw.append((i200)u"\3042" + lineend)
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	99 vwo.append((i200)u"\3042")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	100 self.assertEqual(readalllines("".join(vw), True), "".join(vw))
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	101 self.assertEqual(readalllines("".join(vw), False),"".join(vwo))
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	102
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	103 # Test lines where the first read might end with \r, so the
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	104 # reader has to look ahead whether this is a lone \r or a \r\n
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	105 for size in xrange(80):
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	106 for lineend in u"\n \r\n \r \u2028".split():
34865 b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	107 s = 10(sizeu"a" + lineend + u"xxx\n")
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	108 reader = getreader(s)
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	109 for i in xrange(10):
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	110 self.assertEqual(
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	111 reader.readline(keepends=True),
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	112 size*u"a" + lineend,
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	113 )
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	114 reader = getreader(s)
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	115 for i in xrange(10):
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	116 self.assertEqual(
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	117 reader.readline(keepends=False),
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	118 size*u"a",
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	119 )
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	120
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	121 def test_bug1175396(self):
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	122 s = [
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	123 '<%!--===================================================\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	124 ' BLOG index page: show recent articles,\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	125 ' today\'s articles, or articles of a specific date.\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	126 '========================================================--%>\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	127 '<%@inputencoding="ISO-8859-1"%>\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	128 '<%@pagetemplate=TEMPLATE.y%>\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	129 '<%@import=import frog.util, frog%>\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	130 '<%@import=import frog.objects%>\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	131 '<%@import=from frog.storageerrors import StorageError%>\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	132 '<%\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	133 '\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	134 'import logging\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	135 'log=logging.getLogger("Snakelets.logger")\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	136 '\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	137 '\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	138 'user=self.SessionCtx.user\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	139 'storageEngine=self.SessionCtx.storageEngine\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	140 '\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	141 '\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	142 'def readArticlesFromDate(date, count=None):\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	143 ' entryids=storageEngine.listBlogEntries(date)\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	144 ' entryids.reverse() # descending\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	145 ' if count:\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	146 ' entryids=entryids[:count]\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	147 ' try:\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	148 ' return [ frog.objects.BlogEntry.load(storageEngine, date, Id) for Id in entryids ]\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	149 ' except StorageError,x:\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	150 ' log.error("Error loading articles: "+str(x))\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	151 ' self.abort("cannot load articles")\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	152 '\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	153 'showdate=None\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	154 '\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	155 'arg=self.Request.getArg()\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	156 'if arg=="today":\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	157 ' #-------------------- TODAY\'S ARTICLES\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	158 ' self.write("<h2>Today\'s articles</h2>")\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	159 ' showdate = frog.util.isodatestr() \r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	160 ' entries = readArticlesFromDate(showdate)\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	161 'elif arg=="active":\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	162 ' #-------------------- ACTIVE ARTICLES redirect\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	163 ' self.Yredirect("active.y")\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	164 'elif arg=="login":\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	165 ' #-------------------- LOGIN PAGE redirect\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	166 ' self.Yredirect("login.y")\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	167 'elif arg=="date":\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	168 ' #-------------------- ARTICLES OF A SPECIFIC DATE\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	169 ' showdate = self.Request.getParameter("date")\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	170 ' self.write("<h2>Articles written on %s</h2>"% frog.util.mediumdatestr(showdate))\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	171 ' entries = readArticlesFromDate(showdate)\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	172 'else:\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	173 ' #-------------------- RECENT ARTICLES\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	174 ' self.write("<h2>Recent articles</h2>")\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	175 ' dates=storageEngine.listBlogEntryDates()\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	176 ' if dates:\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	177 ' entries=[]\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	178 ' SHOWAMOUNT=10\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	179 ' for showdate in dates:\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	180 ' entries.extend( readArticlesFromDate(showdate, SHOWAMOUNT-len(entries)) )\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	181 ' if len(entries)>=SHOWAMOUNT:\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	182 ' break\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	183 ' \r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	184 ]
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	185 stream = StringIO.StringIO("".join(s).encode(self.encoding))
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	186 reader = codecs.getreader(self.encoding)(stream)
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	187 for (i, line) in enumerate(reader):
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	188 self.assertEqual(line, s[i])
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	189
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	190 def test_readlinequeue(self):
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	191 q = Queue()
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	192 writer = codecs.getwriter(self.encoding)(q)
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	193 reader = codecs.getreader(self.encoding)(q)
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	194
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	195 # No lineends
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	196 writer.write(u"foo\r")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	197 self.assertEqual(reader.readline(keepends=False), u"foo")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	198 writer.write(u"\nbar\r")
34925 1331c7799640 Update test to the current readline() behaviour. Walter Dörwald <walter@livinglogic.de> parents: 34865 diff changeset	199 self.assertEqual(reader.readline(keepends=False), u"")
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	200 self.assertEqual(reader.readline(keepends=False), u"bar")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	201 writer.write(u"baz")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	202 self.assertEqual(reader.readline(keepends=False), u"baz")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	203 self.assertEqual(reader.readline(keepends=False), u"")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	204
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	205 # Lineends
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	206 writer.write(u"foo\r")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	207 self.assertEqual(reader.readline(keepends=True), u"foo\r")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	208 writer.write(u"\nbar\r")
34925 1331c7799640 Update test to the current readline() behaviour. Walter Dörwald <walter@livinglogic.de> parents: 34865 diff changeset	209 self.assertEqual(reader.readline(keepends=True), u"\n")
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	210 self.assertEqual(reader.readline(keepends=True), u"bar\r")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	211 writer.write(u"baz")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	212 self.assertEqual(reader.readline(keepends=True), u"baz")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	213 self.assertEqual(reader.readline(keepends=True), u"")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	214 writer.write(u"foo\r\n")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	215 self.assertEqual(reader.readline(keepends=True), u"foo\r\n")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	216
34384 a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	217 def test_bug1098990_a(self):
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	218 s1 = u"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy\r\n"
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	219 s2 = u"offending line: ladfj askldfj klasdj fskla dfzaskdj fasklfj laskd fjasklfzzzzaa%whereisthis!!!\r\n"
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	220 s3 = u"next line.\r\n"
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	221
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	222 s = (s1+s2+s3).encode(self.encoding)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	223 stream = StringIO.StringIO(s)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	224 reader = codecs.getreader(self.encoding)(stream)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	225 self.assertEqual(reader.readline(), s1)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	226 self.assertEqual(reader.readline(), s2)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	227 self.assertEqual(reader.readline(), s3)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	228 self.assertEqual(reader.readline(), u"")
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	229
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	230 def test_bug1098990_b(self):
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	231 s1 = u"aaaaaaaaaaaaaaaaaaaaaaaa\r\n"
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	232 s2 = u"bbbbbbbbbbbbbbbbbbbbbbbb\r\n"
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	233 s3 = u"stillokay:bbbbxx\r\n"
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	234 s4 = u"broken!!!!badbad\r\n"
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	235 s5 = u"againokay.\r\n"
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	236
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	237 s = (s1+s2+s3+s4+s5).encode(self.encoding)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	238 stream = StringIO.StringIO(s)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	239 reader = codecs.getreader(self.encoding)(stream)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	240 self.assertEqual(reader.readline(), s1)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	241 self.assertEqual(reader.readline(), s2)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	242 self.assertEqual(reader.readline(), s3)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	243 self.assertEqual(reader.readline(), s4)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	244 self.assertEqual(reader.readline(), s5)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	245 self.assertEqual(reader.readline(), u"")
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	246
42649 c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	247 class UTF32Test(ReadTest):
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	248 encoding = "utf-32"
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	249
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	250 spamle = ('\xff\xfe\x00\x00'
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	251 's\x00\x00\x00p\x00\x00\x00a\x00\x00\x00m\x00\x00\x00'
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	252 's\x00\x00\x00p\x00\x00\x00a\x00\x00\x00m\x00\x00\x00')
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	253 spambe = ('\x00\x00\xfe\xff'
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	254 '\x00\x00\x00s\x00\x00\x00p\x00\x00\x00a\x00\x00\x00m'
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	255 '\x00\x00\x00s\x00\x00\x00p\x00\x00\x00a\x00\x00\x00m')
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	256
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	257 def test_only_one_bom(self):
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	258 _,_,reader,writer = codecs.lookup(self.encoding)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	259 # encode some stream
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	260 s = StringIO.StringIO()
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	261 f = writer(s)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	262 f.write(u"spam")
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	263 f.write(u"spam")
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	264 d = s.getvalue()
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	265 # check whether there is exactly one BOM in it
55002 6768b2350c57 convert usage of fail* to assert* Benjamin Peterson <benjamin@python.org> parents: 53855 diff changeset	266 self.assertTrue(d == self.spamle or d == self.spambe)
42649 c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	267 # try to read it back
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	268 s = StringIO.StringIO(d)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	269 f = reader(s)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	270 self.assertEquals(f.read(), u"spamspam")
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	271
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	272 def test_badbom(self):
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	273 s = StringIO.StringIO(4*"\xff")
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	274 f = codecs.getreader(self.encoding)(s)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	275 self.assertRaises(UnicodeError, f.read)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	276
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	277 s = StringIO.StringIO(8*"\xff")
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	278 f = codecs.getreader(self.encoding)(s)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	279 self.assertRaises(UnicodeError, f.read)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	280
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	281 def test_partial(self):
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	282 self.check_partial(
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	283 u"\x00\xff\u0100\uffff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	284 [
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	285 u"", # first byte of BOM read
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	286 u"", # second byte of BOM read
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	287 u"", # third byte of BOM read
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	288 u"", # fourth byte of BOM read => byteorder known
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	289 u"",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	290 u"",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	291 u"",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	292 u"\x00",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	293 u"\x00",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	294 u"\x00",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	295 u"\x00",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	296 u"\x00\xff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	297 u"\x00\xff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	298 u"\x00\xff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	299 u"\x00\xff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	300 u"\x00\xff\u0100",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	301 u"\x00\xff\u0100",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	302 u"\x00\xff\u0100",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	303 u"\x00\xff\u0100",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	304 u"\x00\xff\u0100\uffff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	305 ]
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	306 )
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	307
56033 38a52b672429 Issue #6922: Fix an infinite loop when trying to decode an invalid Georg Brandl <georg@python.org> parents: 55002 diff changeset	308 def test_handlers(self):
38a52b672429 Issue #6922: Fix an infinite loop when trying to decode an invalid Georg Brandl <georg@python.org> parents: 55002 diff changeset	309 self.assertEqual((u'\ufffd', 1),
38a52b672429 Issue #6922: Fix an infinite loop when trying to decode an invalid Georg Brandl <georg@python.org> parents: 55002 diff changeset	310 codecs.utf_32_decode('\x01', 'replace', True))
38a52b672429 Issue #6922: Fix an infinite loop when trying to decode an invalid Georg Brandl <georg@python.org> parents: 55002 diff changeset	311 self.assertEqual((u'', 1),
38a52b672429 Issue #6922: Fix an infinite loop when trying to decode an invalid Georg Brandl <georg@python.org> parents: 55002 diff changeset	312 codecs.utf_32_decode('\x01', 'ignore', True))
38a52b672429 Issue #6922: Fix an infinite loop when trying to decode an invalid Georg Brandl <georg@python.org> parents: 55002 diff changeset	313
42649 c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	314 def test_errors(self):
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	315 self.assertRaises(UnicodeDecodeError, codecs.utf_32_decode,
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	316 "\xff", "strict", True)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	317
62255 98245464b665 Issue #8941: decoding big endian UTF-32 data in UCS-2 builds could crash Antoine Pitrou <solipsis@pitrou.net> parents: 61884 diff changeset	318 def test_issue8941(self):
98245464b665 Issue #8941: decoding big endian UTF-32 data in UCS-2 builds could crash Antoine Pitrou <solipsis@pitrou.net> parents: 61884 diff changeset	319 # Issue #8941: insufficient result allocation when decoding into
98245464b665 Issue #8941: decoding big endian UTF-32 data in UCS-2 builds could crash Antoine Pitrou <solipsis@pitrou.net> parents: 61884 diff changeset	320 # surrogate pairs on UCS-2 builds.
98245464b665 Issue #8941: decoding big endian UTF-32 data in UCS-2 builds could crash Antoine Pitrou <solipsis@pitrou.net> parents: 61884 diff changeset	321 encoded_le = '\xff\xfe\x00\x00' + '\x00\x00\x01\x00' * 1024
98245464b665 Issue #8941: decoding big endian UTF-32 data in UCS-2 builds could crash Antoine Pitrou <solipsis@pitrou.net> parents: 61884 diff changeset	322 self.assertEqual(u'\U00010000' * 1024,
98245464b665 Issue #8941: decoding big endian UTF-32 data in UCS-2 builds could crash Antoine Pitrou <solipsis@pitrou.net> parents: 61884 diff changeset	323 codecs.utf_32_decode(encoded_le)[0])
98245464b665 Issue #8941: decoding big endian UTF-32 data in UCS-2 builds could crash Antoine Pitrou <solipsis@pitrou.net> parents: 61884 diff changeset	324 encoded_be = '\x00\x00\xfe\xff' + '\x00\x01\x00\x00' * 1024
98245464b665 Issue #8941: decoding big endian UTF-32 data in UCS-2 builds could crash Antoine Pitrou <solipsis@pitrou.net> parents: 61884 diff changeset	325 self.assertEqual(u'\U00010000' * 1024,
98245464b665 Issue #8941: decoding big endian UTF-32 data in UCS-2 builds could crash Antoine Pitrou <solipsis@pitrou.net> parents: 61884 diff changeset	326 codecs.utf_32_decode(encoded_be)[0])
98245464b665 Issue #8941: decoding big endian UTF-32 data in UCS-2 builds could crash Antoine Pitrou <solipsis@pitrou.net> parents: 61884 diff changeset	327
42649 c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	328 class UTF32LETest(ReadTest):
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	329 encoding = "utf-32-le"
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	330
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	331 def test_partial(self):
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	332 self.check_partial(
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	333 u"\x00\xff\u0100\uffff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	334 [
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	335 u"",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	336 u"",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	337 u"",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	338 u"\x00",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	339 u"\x00",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	340 u"\x00",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	341 u"\x00",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	342 u"\x00\xff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	343 u"\x00\xff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	344 u"\x00\xff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	345 u"\x00\xff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	346 u"\x00\xff\u0100",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	347 u"\x00\xff\u0100",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	348 u"\x00\xff\u0100",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	349 u"\x00\xff\u0100",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	350 u"\x00\xff\u0100\uffff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	351 ]
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	352 )
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	353
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	354 def test_simple(self):
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	355 self.assertEqual(u"\U00010203".encode(self.encoding), "\x03\x02\x01\x00")
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	356
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	357 def test_errors(self):
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	358 self.assertRaises(UnicodeDecodeError, codecs.utf_32_le_decode,
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	359 "\xff", "strict", True)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	360
62255 98245464b665 Issue #8941: decoding big endian UTF-32 data in UCS-2 builds could crash Antoine Pitrou <solipsis@pitrou.net> parents: 61884 diff changeset	361 def test_issue8941(self):
98245464b665 Issue #8941: decoding big endian UTF-32 data in UCS-2 builds could crash Antoine Pitrou <solipsis@pitrou.net> parents: 61884 diff changeset	362 # Issue #8941: insufficient result allocation when decoding into
98245464b665 Issue #8941: decoding big endian UTF-32 data in UCS-2 builds could crash Antoine Pitrou <solipsis@pitrou.net> parents: 61884 diff changeset	363 # surrogate pairs on UCS-2 builds.
98245464b665 Issue #8941: decoding big endian UTF-32 data in UCS-2 builds could crash Antoine Pitrou <solipsis@pitrou.net> parents: 61884 diff changeset	364 encoded = '\x00\x00\x01\x00' * 1024
98245464b665 Issue #8941: decoding big endian UTF-32 data in UCS-2 builds could crash Antoine Pitrou <solipsis@pitrou.net> parents: 61884 diff changeset	365 self.assertEqual(u'\U00010000' * 1024,
98245464b665 Issue #8941: decoding big endian UTF-32 data in UCS-2 builds could crash Antoine Pitrou <solipsis@pitrou.net> parents: 61884 diff changeset	366 codecs.utf_32_le_decode(encoded)[0])
98245464b665 Issue #8941: decoding big endian UTF-32 data in UCS-2 builds could crash Antoine Pitrou <solipsis@pitrou.net> parents: 61884 diff changeset	367
42649 c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	368 class UTF32BETest(ReadTest):
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	369 encoding = "utf-32-be"
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	370
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	371 def test_partial(self):
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	372 self.check_partial(
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	373 u"\x00\xff\u0100\uffff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	374 [
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	375 u"",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	376 u"",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	377 u"",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	378 u"\x00",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	379 u"\x00",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	380 u"\x00",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	381 u"\x00",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	382 u"\x00\xff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	383 u"\x00\xff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	384 u"\x00\xff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	385 u"\x00\xff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	386 u"\x00\xff\u0100",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	387 u"\x00\xff\u0100",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	388 u"\x00\xff\u0100",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	389 u"\x00\xff\u0100",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	390 u"\x00\xff\u0100\uffff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	391 ]
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	392 )
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	393
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	394 def test_simple(self):
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	395 self.assertEqual(u"\U00010203".encode(self.encoding), "\x00\x01\x02\x03")
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	396
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	397 def test_errors(self):
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	398 self.assertRaises(UnicodeDecodeError, codecs.utf_32_be_decode,
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	399 "\xff", "strict", True)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	400
62255 98245464b665 Issue #8941: decoding big endian UTF-32 data in UCS-2 builds could crash Antoine Pitrou <solipsis@pitrou.net> parents: 61884 diff changeset	401 def test_issue8941(self):
98245464b665 Issue #8941: decoding big endian UTF-32 data in UCS-2 builds could crash Antoine Pitrou <solipsis@pitrou.net> parents: 61884 diff changeset	402 # Issue #8941: insufficient result allocation when decoding into
98245464b665 Issue #8941: decoding big endian UTF-32 data in UCS-2 builds could crash Antoine Pitrou <solipsis@pitrou.net> parents: 61884 diff changeset	403 # surrogate pairs on UCS-2 builds.
98245464b665 Issue #8941: decoding big endian UTF-32 data in UCS-2 builds could crash Antoine Pitrou <solipsis@pitrou.net> parents: 61884 diff changeset	404 encoded = '\x00\x01\x00\x00' * 1024
98245464b665 Issue #8941: decoding big endian UTF-32 data in UCS-2 builds could crash Antoine Pitrou <solipsis@pitrou.net> parents: 61884 diff changeset	405 self.assertEqual(u'\U00010000' * 1024,
98245464b665 Issue #8941: decoding big endian UTF-32 data in UCS-2 builds could crash Antoine Pitrou <solipsis@pitrou.net> parents: 61884 diff changeset	406 codecs.utf_32_be_decode(encoded)[0])
98245464b665 Issue #8941: decoding big endian UTF-32 data in UCS-2 builds could crash Antoine Pitrou <solipsis@pitrou.net> parents: 61884 diff changeset	407
98245464b665 Issue #8941: decoding big endian UTF-32 data in UCS-2 builds could crash Antoine Pitrou <solipsis@pitrou.net> parents: 61884 diff changeset	408
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	409 class UTF16Test(ReadTest):
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	410 encoding = "utf-16"
18163 f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	411
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	412 spamle = '\xff\xfes\x00p\x00a\x00m\x00s\x00p\x00a\x00m\x00'
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	413 spambe = '\xfe\xff\x00s\x00p\x00a\x00m\x00s\x00p\x00a\x00m'
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	414
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	415 def test_only_one_bom(self):
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	416 _,_,reader,writer = codecs.lookup(self.encoding)
18163 f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	417 # encode some stream
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	418 s = StringIO.StringIO()
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	419 f = writer(s)
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	420 f.write(u"spam")
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	421 f.write(u"spam")
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	422 d = s.getvalue()
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	423 # check whether there is exactly one BOM in it
55002 6768b2350c57 convert usage of fail* to assert* Benjamin Peterson <benjamin@python.org> parents: 53855 diff changeset	424 self.assertTrue(d == self.spamle or d == self.spambe)
18163 f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	425 # try to read it back
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	426 s = StringIO.StringIO(d)
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	427 f = reader(s)
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	428 self.assertEquals(f.read(), u"spamspam")
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	429
34538 37b2f11c1a00 Add a test for UTF-16 reading where the byte sequence doesn't start with Walter Dörwald <walter@livinglogic.de> parents: 34384 diff changeset	430 def test_badbom(self):
37b2f11c1a00 Add a test for UTF-16 reading where the byte sequence doesn't start with Walter Dörwald <walter@livinglogic.de> parents: 34384 diff changeset	431 s = StringIO.StringIO("\xff\xff")
34562 7a1277fad04e Fix stupid typo: Don't read from a writer. Walter Dörwald <walter@livinglogic.de> parents: 34538 diff changeset	432 f = codecs.getreader(self.encoding)(s)
34538 37b2f11c1a00 Add a test for UTF-16 reading where the byte sequence doesn't start with Walter Dörwald <walter@livinglogic.de> parents: 34384 diff changeset	433 self.assertRaises(UnicodeError, f.read)
37b2f11c1a00 Add a test for UTF-16 reading where the byte sequence doesn't start with Walter Dörwald <walter@livinglogic.de> parents: 34384 diff changeset	434
37b2f11c1a00 Add a test for UTF-16 reading where the byte sequence doesn't start with Walter Dörwald <walter@livinglogic.de> parents: 34384 diff changeset	435 s = StringIO.StringIO("\xff\xff\xff\xff")
34562 7a1277fad04e Fix stupid typo: Don't read from a writer. Walter Dörwald <walter@livinglogic.de> parents: 34538 diff changeset	436 f = codecs.getreader(self.encoding)(s)
34538 37b2f11c1a00 Add a test for UTF-16 reading where the byte sequence doesn't start with Walter Dörwald <walter@livinglogic.de> parents: 34384 diff changeset	437 self.assertRaises(UnicodeError, f.read)
37b2f11c1a00 Add a test for UTF-16 reading where the byte sequence doesn't start with Walter Dörwald <walter@livinglogic.de> parents: 34384 diff changeset	438
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	439 def test_partial(self):
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	440 self.check_partial(
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	441 u"\x00\xff\u0100\uffff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	442 [
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	443 u"", # first byte of BOM read
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	444 u"", # second byte of BOM read => byteorder known
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	445 u"",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	446 u"\x00",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	447 u"\x00",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	448 u"\x00\xff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	449 u"\x00\xff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	450 u"\x00\xff\u0100",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	451 u"\x00\xff\u0100",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	452 u"\x00\xff\u0100\uffff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	453 ]
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	454 )
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	455
56033 38a52b672429 Issue #6922: Fix an infinite loop when trying to decode an invalid Georg Brandl <georg@python.org> parents: 55002 diff changeset	456 def test_handlers(self):
38a52b672429 Issue #6922: Fix an infinite loop when trying to decode an invalid Georg Brandl <georg@python.org> parents: 55002 diff changeset	457 self.assertEqual((u'\ufffd', 1),
38a52b672429 Issue #6922: Fix an infinite loop when trying to decode an invalid Georg Brandl <georg@python.org> parents: 55002 diff changeset	458 codecs.utf_16_decode('\x01', 'replace', True))
38a52b672429 Issue #6922: Fix an infinite loop when trying to decode an invalid Georg Brandl <georg@python.org> parents: 55002 diff changeset	459 self.assertEqual((u'', 1),
38a52b672429 Issue #6922: Fix an infinite loop when trying to decode an invalid Georg Brandl <georg@python.org> parents: 55002 diff changeset	460 codecs.utf_16_decode('\x01', 'ignore', True))
38a52b672429 Issue #6922: Fix an infinite loop when trying to decode an invalid Georg Brandl <georg@python.org> parents: 55002 diff changeset	461
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	462 def test_errors(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	463 self.assertRaises(UnicodeDecodeError, codecs.utf_16_decode, "\xff", "strict", True)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	464
59147 7e3c47354da3 #691291: codecs.open() should not convert end of lines on reading and writing. Florent Xicluna <florent.xicluna@gmail.com> parents: 58472 diff changeset	465 def test_bug691291(self):
7e3c47354da3 #691291: codecs.open() should not convert end of lines on reading and writing. Florent Xicluna <florent.xicluna@gmail.com> parents: 58472 diff changeset	466 # Files are always opened in binary mode, even if no binary mode was
7e3c47354da3 #691291: codecs.open() should not convert end of lines on reading and writing. Florent Xicluna <florent.xicluna@gmail.com> parents: 58472 diff changeset	467 # specified. This means that no automatic conversion of '\n' is done
7e3c47354da3 #691291: codecs.open() should not convert end of lines on reading and writing. Florent Xicluna <florent.xicluna@gmail.com> parents: 58472 diff changeset	468 # on reading and writing.
7e3c47354da3 #691291: codecs.open() should not convert end of lines on reading and writing. Florent Xicluna <florent.xicluna@gmail.com> parents: 58472 diff changeset	469 s1 = u'Hello\r\nworld\r\n'
7e3c47354da3 #691291: codecs.open() should not convert end of lines on reading and writing. Florent Xicluna <florent.xicluna@gmail.com> parents: 58472 diff changeset	470
7e3c47354da3 #691291: codecs.open() should not convert end of lines on reading and writing. Florent Xicluna <florent.xicluna@gmail.com> parents: 58472 diff changeset	471 s = s1.encode(self.encoding)
7e3c47354da3 #691291: codecs.open() should not convert end of lines on reading and writing. Florent Xicluna <florent.xicluna@gmail.com> parents: 58472 diff changeset	472 try:
7e3c47354da3 #691291: codecs.open() should not convert end of lines on reading and writing. Florent Xicluna <florent.xicluna@gmail.com> parents: 58472 diff changeset	473 with open(test_support.TESTFN, 'wb') as fp:
7e3c47354da3 #691291: codecs.open() should not convert end of lines on reading and writing. Florent Xicluna <florent.xicluna@gmail.com> parents: 58472 diff changeset	474 fp.write(s)
7e3c47354da3 #691291: codecs.open() should not convert end of lines on reading and writing. Florent Xicluna <florent.xicluna@gmail.com> parents: 58472 diff changeset	475 with codecs.open(test_support.TESTFN, 'U', encoding=self.encoding) as reader:
7e3c47354da3 #691291: codecs.open() should not convert end of lines on reading and writing. Florent Xicluna <florent.xicluna@gmail.com> parents: 58472 diff changeset	476 self.assertEqual(reader.read(), s1)
7e3c47354da3 #691291: codecs.open() should not convert end of lines on reading and writing. Florent Xicluna <florent.xicluna@gmail.com> parents: 58472 diff changeset	477 finally:
7e3c47354da3 #691291: codecs.open() should not convert end of lines on reading and writing. Florent Xicluna <florent.xicluna@gmail.com> parents: 58472 diff changeset	478 test_support.unlink(test_support.TESTFN)
7e3c47354da3 #691291: codecs.open() should not convert end of lines on reading and writing. Florent Xicluna <florent.xicluna@gmail.com> parents: 58472 diff changeset	479
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	480 class UTF16LETest(ReadTest):
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	481 encoding = "utf-16-le"
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	482
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	483 def test_partial(self):
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	484 self.check_partial(
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	485 u"\x00\xff\u0100\uffff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	486 [
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	487 u"",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	488 u"\x00",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	489 u"\x00",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	490 u"\x00\xff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	491 u"\x00\xff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	492 u"\x00\xff\u0100",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	493 u"\x00\xff\u0100",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	494 u"\x00\xff\u0100\uffff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	495 ]
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	496 )
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	497
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	498 def test_errors(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	499 self.assertRaises(UnicodeDecodeError, codecs.utf_16_le_decode, "\xff", "strict", True)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	500
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	501 class UTF16BETest(ReadTest):
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	502 encoding = "utf-16-be"
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	503
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	504 def test_partial(self):
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	505 self.check_partial(
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	506 u"\x00\xff\u0100\uffff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	507 [
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	508 u"",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	509 u"\x00",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	510 u"\x00",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	511 u"\x00\xff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	512 u"\x00\xff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	513 u"\x00\xff\u0100",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	514 u"\x00\xff\u0100",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	515 u"\x00\xff\u0100\uffff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	516 ]
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	517 )
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	518
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	519 def test_errors(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	520 self.assertRaises(UnicodeDecodeError, codecs.utf_16_be_decode, "\xff", "strict", True)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	521
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	522 class UTF8Test(ReadTest):
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	523 encoding = "utf-8"
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	524
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	525 def test_partial(self):
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	526 self.check_partial(
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	527 u"\x00\xff\u07ff\u0800\uffff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	528 [
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	529 u"\x00",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	530 u"\x00",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	531 u"\x00\xff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	532 u"\x00\xff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	533 u"\x00\xff\u07ff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	534 u"\x00\xff\u07ff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	535 u"\x00\xff\u07ff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	536 u"\x00\xff\u07ff\u0800",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	537 u"\x00\xff\u07ff\u0800",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	538 u"\x00\xff\u07ff\u0800",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	539 u"\x00\xff\u07ff\u0800\uffff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	540 ]
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	541 )
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	542
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	543 class UTF7Test(ReadTest):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	544 encoding = "utf-7"
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	545
43932 e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls. Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 43905 diff changeset	546 def test_partial(self):
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls. Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 43905 diff changeset	547 self.check_partial(
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls. Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 43905 diff changeset	548 u"a+-b",
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls. Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 43905 diff changeset	549 [
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls. Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 43905 diff changeset	550 u"a",
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls. Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 43905 diff changeset	551 u"a",
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls. Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 43905 diff changeset	552 u"a+",
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls. Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 43905 diff changeset	553 u"a+-",
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls. Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 43905 diff changeset	554 u"a+-b",
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls. Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 43905 diff changeset	555 ]
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls. Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 43905 diff changeset	556 )
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	557
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	558 class UTF16ExTest(unittest.TestCase):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	559
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	560 def test_errors(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	561 self.assertRaises(UnicodeDecodeError, codecs.utf_16_ex_decode, "\xff", "strict", 0, True)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	562
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	563 def test_bad_args(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	564 self.assertRaises(TypeError, codecs.utf_16_ex_decode)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	565
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	566 class ReadBufferTest(unittest.TestCase):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	567
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	568 def test_array(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	569 import array
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	570 self.assertEqual(
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	571 codecs.readbuffer_encode(array.array("c", "spam")),
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	572 ("spam", 4)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	573 )
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	574
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	575 def test_empty(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	576 self.assertEqual(codecs.readbuffer_encode(""), ("", 0))
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	577
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	578 def test_bad_args(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	579 self.assertRaises(TypeError, codecs.readbuffer_encode)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	580 self.assertRaises(TypeError, codecs.readbuffer_encode, 42)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	581
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	582 class CharBufferTest(unittest.TestCase):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	583
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	584 def test_string(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	585 self.assertEqual(codecs.charbuffer_encode("spam"), ("spam", 4))
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	586
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	587 def test_empty(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	588 self.assertEqual(codecs.charbuffer_encode(""), ("", 0))
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	589
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	590 def test_bad_args(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	591 self.assertRaises(TypeError, codecs.charbuffer_encode)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	592 self.assertRaises(TypeError, codecs.charbuffer_encode, 42)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	593
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	594 class UTF8SigTest(ReadTest):
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	595 encoding = "utf-8-sig"
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	596
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	597 def test_partial(self):
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	598 self.check_partial(
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	599 u"\ufeff\x00\xff\u07ff\u0800\uffff",
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	600 [
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	601 u"",
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	602 u"",
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	603 u"", # First BOM has been read and skipped
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	604 u"",
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	605 u"",
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	606 u"\ufeff", # Second BOM has been read and emitted
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	607 u"\ufeff\x00", # "\x00" read and emitted
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	608 u"\ufeff\x00", # First byte of encoded u"\xff" read
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	609 u"\ufeff\x00\xff", # Second byte of encoded u"\xff" read
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	610 u"\ufeff\x00\xff", # First byte of encoded u"\u07ff" read
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	611 u"\ufeff\x00\xff\u07ff", # Second byte of encoded u"\u07ff" read
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	612 u"\ufeff\x00\xff\u07ff",
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	613 u"\ufeff\x00\xff\u07ff",
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	614 u"\ufeff\x00\xff\u07ff\u0800",
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	615 u"\ufeff\x00\xff\u07ff\u0800",
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	616 u"\ufeff\x00\xff\u07ff\u0800",
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	617 u"\ufeff\x00\xff\u07ff\u0800\uffff",
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	618 ]
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	619 )
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	620
40386 a3e0f1e11082 Change decode() so that it works with a buffer (i.e. unicode(..., 'utf-8-sig')) Walter Dörwald <walter@livinglogic.de> parents: 40284 diff changeset	621 def test_bug1601501(self):
a3e0f1e11082 Change decode() so that it works with a buffer (i.e. unicode(..., 'utf-8-sig')) Walter Dörwald <walter@livinglogic.de> parents: 40284 diff changeset	622 # SF bug #1601501: check that the codec works with a buffer
a3e0f1e11082 Change decode() so that it works with a buffer (i.e. unicode(..., 'utf-8-sig')) Walter Dörwald <walter@livinglogic.de> parents: 40284 diff changeset	623 unicode("\xef\xbb\xbf", "utf-8-sig")
a3e0f1e11082 Change decode() so that it works with a buffer (i.e. unicode(..., 'utf-8-sig')) Walter Dörwald <walter@livinglogic.de> parents: 40284 diff changeset	624
41445 8447314f500a Fix utf-8-sig incremental decoder, which didn't recognise a BOM when the Walter Dörwald <walter@livinglogic.de> parents: 40386 diff changeset	625 def test_bom(self):
8447314f500a Fix utf-8-sig incremental decoder, which didn't recognise a BOM when the Walter Dörwald <walter@livinglogic.de> parents: 40386 diff changeset	626 d = codecs.getincrementaldecoder("utf-8-sig")()
8447314f500a Fix utf-8-sig incremental decoder, which didn't recognise a BOM when the Walter Dörwald <walter@livinglogic.de> parents: 40386 diff changeset	627 s = u"spam"
8447314f500a Fix utf-8-sig incremental decoder, which didn't recognise a BOM when the Walter Dörwald <walter@livinglogic.de> parents: 40386 diff changeset	628 self.assertEqual(d.decode(s.encode("utf-8-sig")), s)
8447314f500a Fix utf-8-sig incremental decoder, which didn't recognise a BOM when the Walter Dörwald <walter@livinglogic.de> parents: 40386 diff changeset	629
43905 0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	630 def test_stream_bom(self):
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	631 unistring = u"ABC\u00A1\u2200XYZ"
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	632 bytestring = codecs.BOM_UTF8 + "ABC\xC2\xA1\xE2\x88\x80XYZ"
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	633
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	634 reader = codecs.getreader("utf-8-sig")
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	635 for sizehint in [None] + range(1, 11) + \
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	636 [64, 128, 256, 512, 1024]:
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	637 istream = reader(StringIO.StringIO(bytestring))
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	638 ostream = StringIO.StringIO()
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	639 while 1:
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	640 if sizehint is not None:
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	641 data = istream.read(sizehint)
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	642 else:
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	643 data = istream.read()
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	644
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	645 if not data:
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	646 break
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	647 ostream.write(data)
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	648
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	649 got = ostream.getvalue()
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	650 self.assertEqual(got, unistring)
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	651
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	652 def test_stream_bare(self):
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	653 unistring = u"ABC\u00A1\u2200XYZ"
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	654 bytestring = "ABC\xC2\xA1\xE2\x88\x80XYZ"
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	655
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	656 reader = codecs.getreader("utf-8-sig")
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	657 for sizehint in [None] + range(1, 11) + \
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	658 [64, 128, 256, 512, 1024]:
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	659 istream = reader(StringIO.StringIO(bytestring))
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	660 ostream = StringIO.StringIO()
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	661 while 1:
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	662 if sizehint is not None:
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	663 data = istream.read(sizehint)
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	664 else:
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	665 data = istream.read()
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	666
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	667 if not data:
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	668 break
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	669 ostream.write(data)
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	670
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	671 got = ostream.getvalue()
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	672 self.assertEqual(got, unistring)
0e897905ee94 Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode()) Walter Dörwald <walter@livinglogic.de> parents: 43903 diff changeset	673
25100 bc590beadf0b Check whether a string resize is necessary at the end Walter Dörwald <walter@livinglogic.de> parents: 24364 diff changeset	674 class EscapeDecodeTest(unittest.TestCase):
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	675 def test_empty(self):
25100 bc590beadf0b Check whether a string resize is necessary at the end Walter Dörwald <walter@livinglogic.de> parents: 24364 diff changeset	676 self.assertEquals(codecs.escape_decode(""), ("", 0))
bc590beadf0b Check whether a string resize is necessary at the end Walter Dörwald <walter@livinglogic.de> parents: 24364 diff changeset	677
27680 b19aaaeb6d64 Fix for [ 543344 ] Interpreter crashes when recoding; suggested Marc-André Lemburg <mal@egenix.com> parents: 25100 diff changeset	678 class RecodingTest(unittest.TestCase):
b19aaaeb6d64 Fix for [ 543344 ] Interpreter crashes when recoding; suggested Marc-André Lemburg <mal@egenix.com> parents: 25100 diff changeset	679 def test_recoding(self):
b19aaaeb6d64 Fix for [ 543344 ] Interpreter crashes when recoding; suggested Marc-André Lemburg <mal@egenix.com> parents: 25100 diff changeset	680 f = StringIO.StringIO()
b19aaaeb6d64 Fix for [ 543344 ] Interpreter crashes when recoding; suggested Marc-André Lemburg <mal@egenix.com> parents: 25100 diff changeset	681 f2 = codecs.EncodedFile(f, "unicode_internal", "utf-8")
b19aaaeb6d64 Fix for [ 543344 ] Interpreter crashes when recoding; suggested Marc-André Lemburg <mal@egenix.com> parents: 25100 diff changeset	682 f2.write(u"a")
b19aaaeb6d64 Fix for [ 543344 ] Interpreter crashes when recoding; suggested Marc-André Lemburg <mal@egenix.com> parents: 25100 diff changeset	683 f2.close()
b19aaaeb6d64 Fix for [ 543344 ] Interpreter crashes when recoding; suggested Marc-André Lemburg <mal@egenix.com> parents: 25100 diff changeset	684 # Python used to crash on this at exit because of a refcount
b19aaaeb6d64 Fix for [ 543344 ] Interpreter crashes when recoding; suggested Marc-André Lemburg <mal@egenix.com> parents: 25100 diff changeset	685 # bug in _codecsmodule.c
20013 e3184e46303b Change the PyUnit-based tests to use the test_main() approach. This Fred Drake <fdrake@acm.org> parents: 18163 diff changeset	686
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	687 # From RFC 3492
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	688 punycode_testcases = [
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	689 # A Arabic (Egyptian):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	690 (u"\u0644\u064A\u0647\u0645\u0627\u0628\u062A\u0643\u0644"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	691 u"\u0645\u0648\u0634\u0639\u0631\u0628\u064A\u061F",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	692 "egbpdaj6bu4bxfgehfvwxn"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	693 # B Chinese (simplified):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	694 (u"\u4ED6\u4EEC\u4E3A\u4EC0\u4E48\u4E0D\u8BF4\u4E2D\u6587",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	695 "ihqwcrb4cv8a8dqg056pqjye"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	696 # C Chinese (traditional):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	697 (u"\u4ED6\u5011\u7232\u4EC0\u9EBD\u4E0D\u8AAA\u4E2D\u6587",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	698 "ihqwctvzc91f659drss3x8bo0yb"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	699 # D Czech: Pro<ccaron>prost<ecaron>nemluv<iacute><ccaron>esky
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	700 (u"\u0050\u0072\u006F\u010D\u0070\u0072\u006F\u0073\u0074"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	701 u"\u011B\u006E\u0065\u006D\u006C\u0075\u0076\u00ED\u010D"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	702 u"\u0065\u0073\u006B\u0079",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	703 "Proprostnemluvesky-uyb24dma41a"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	704 # E Hebrew:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	705 (u"\u05DC\u05DE\u05D4\u05D4\u05DD\u05E4\u05E9\u05D5\u05D8"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	706 u"\u05DC\u05D0\u05DE\u05D3\u05D1\u05E8\u05D9\u05DD\u05E2"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	707 u"\u05D1\u05E8\u05D9\u05EA",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	708 "4dbcagdahymbxekheh6e0a7fei0b"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	709 # F Hindi (Devanagari):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	710 (u"\u092F\u0939\u0932\u094B\u0917\u0939\u093F\u0928\u094D"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	711 u"\u0926\u0940\u0915\u094D\u092F\u094B\u0902\u0928\u0939"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	712 u"\u0940\u0902\u092C\u094B\u0932\u0938\u0915\u0924\u0947"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	713 u"\u0939\u0948\u0902",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	714 "i1baa7eci9glrd9b2ae1bj0hfcgg6iyaf8o0a1dig0cd"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	715
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	716 #(G) Japanese (kanji and hiragana):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	717 (u"\u306A\u305C\u307F\u3093\u306A\u65E5\u672C\u8A9E\u3092"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	718 u"\u8A71\u3057\u3066\u304F\u308C\u306A\u3044\u306E\u304B",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	719 "n8jok5ay5dzabd5bym9f0cm5685rrjetr6pdxa"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	720
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	721 # (H) Korean (Hangul syllables):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	722 (u"\uC138\uACC4\uC758\uBAA8\uB4E0\uC0AC\uB78C\uB4E4\uC774"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	723 u"\uD55C\uAD6D\uC5B4\uB97C\uC774\uD574\uD55C\uB2E4\uBA74"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	724 u"\uC5BC\uB9C8\uB098\uC88B\uC744\uAE4C",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	725 "989aomsvi5e83db1d2a355cv1e0vak1dwrv93d5xbh15a0dt30a5j"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	726 "psd879ccm6fea98c"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	727
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	728 # (I) Russian (Cyrillic):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	729 (u"\u043F\u043E\u0447\u0435\u043C\u0443\u0436\u0435\u043E"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	730 u"\u043D\u0438\u043D\u0435\u0433\u043E\u0432\u043E\u0440"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	731 u"\u044F\u0442\u043F\u043E\u0440\u0443\u0441\u0441\u043A"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	732 u"\u0438",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	733 "b1abfaaepdrnnbgefbaDotcwatmq2g4l"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	734
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	735 # (J) Spanish: Porqu<eacute>nopuedensimplementehablarenEspa<ntilde>ol
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	736 (u"\u0050\u006F\u0072\u0071\u0075\u00E9\u006E\u006F\u0070"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	737 u"\u0075\u0065\u0064\u0065\u006E\u0073\u0069\u006D\u0070"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	738 u"\u006C\u0065\u006D\u0065\u006E\u0074\u0065\u0068\u0061"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	739 u"\u0062\u006C\u0061\u0072\u0065\u006E\u0045\u0073\u0070"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	740 u"\u0061\u00F1\u006F\u006C",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	741 "PorqunopuedensimplementehablarenEspaol-fmd56a"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	742
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	743 # (K) Vietnamese:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	744 # T<adotbelow>isaoh<odotbelow>kh<ocirc>ngth<ecirchookabove>ch\
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	745 # <ihookabove>n<oacute>iti<ecircacute>ngVi<ecircdotbelow>t
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	746 (u"\u0054\u1EA1\u0069\u0073\u0061\u006F\u0068\u1ECD\u006B"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	747 u"\u0068\u00F4\u006E\u0067\u0074\u0068\u1EC3\u0063\u0068"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	748 u"\u1EC9\u006E\u00F3\u0069\u0074\u0069\u1EBF\u006E\u0067"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	749 u"\u0056\u0069\u1EC7\u0074",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	750 "TisaohkhngthchnitingVit-kjcr8268qyxafd2f1b9g"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	751
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	752 #(L) 3<nen>B<gumi><kinpachi><sensei>
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	753 (u"\u0033\u5E74\u0042\u7D44\u91D1\u516B\u5148\u751F",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	754 "3B-ww4c5e180e575a65lsy2b"),
28774 c9bee11eaa8b Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 28678 diff changeset	755
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	756 # (M) <amuro><namie>-with-SUPER-MONKEYS
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	757 (u"\u5B89\u5BA4\u5948\u7F8E\u6075\u002D\u0077\u0069\u0074"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	758 u"\u0068\u002D\u0053\u0055\u0050\u0045\u0052\u002D\u004D"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	759 u"\u004F\u004E\u004B\u0045\u0059\u0053",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	760 "-with-SUPER-MONKEYS-pc58ag80a8qai00g7n9n"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	761
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	762 # (N) Hello-Another-Way-<sorezore><no><basho>
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	763 (u"\u0048\u0065\u006C\u006C\u006F\u002D\u0041\u006E\u006F"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	764 u"\u0074\u0068\u0065\u0072\u002D\u0057\u0061\u0079\u002D"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	765 u"\u305D\u308C\u305E\u308C\u306E\u5834\u6240",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	766 "Hello-Another-Way--fc4qua05auwb3674vfr0b"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	767
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	768 # (O) <hitotsu><yane><no><shita>2
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	769 (u"\u3072\u3068\u3064\u5C4B\u6839\u306E\u4E0B\u0032",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	770 "2-u9tlzr9756bt3uc0v"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	771
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	772 # (P) Maji<de>Koi<suru>5<byou><mae>
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	773 (u"\u004D\u0061\u006A\u0069\u3067\u004B\u006F\u0069\u3059"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	774 u"\u308B\u0035\u79D2\u524D",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	775 "MajiKoi5-783gue6qz075azm5e"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	776
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	777 # (Q) <pafii>de<runba>
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	778 (u"\u30D1\u30D5\u30A3\u30FC\u0064\u0065\u30EB\u30F3\u30D0",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	779 "de-jg4avhby1noc0d"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	780
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	781 # (R) <sono><supiido><de>
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	782 (u"\u305D\u306E\u30B9\u30D4\u30FC\u30C9\u3067",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	783 "d9juau41awczczp"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	784
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	785 # (S) -> $1.00 <-
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	786 (u"\u002D\u003E\u0020\u0024\u0031\u002E\u0030\u0030\u0020"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	787 u"\u003C\u002D",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	788 "-> $1.00 <--")
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	789 ]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	790
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	791 for i in punycode_testcases:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	792 if len(i)!=2:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	793 print repr(i)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	794
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	795 class PunycodeTest(unittest.TestCase):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	796 def test_encode(self):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	797 for uni, puny in punycode_testcases:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	798 # Need to convert both strings to lower case, since
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	799 # some of the extended encodings use upper case, but our
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	800 # code produces only lower case. Converting just puny to
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	801 # lower is also insufficient, since some of the input characters
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	802 # are upper case.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	803 self.assertEquals(uni.encode("punycode").lower(), puny.lower())
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	804
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	805 def test_decode(self):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	806 for uni, puny in punycode_testcases:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	807 self.assertEquals(uni, puny.decode("punycode"))
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	808
35484 4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	809 class UnicodeInternalTest(unittest.TestCase):
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	810 def test_bug1251300(self):
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	811 # Decoding with unicode_internal used to not correctly handle "code
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	812 # points" above 0x10ffff on UCS-4 builds.
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	813 if sys.maxunicode > 0xffff:
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	814 ok = [
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	815 ("\x00\x10\xff\xff", u"\U0010ffff"),
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	816 ("\x00\x00\x01\x01", u"\U00000101"),
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	817 ("", u""),
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	818 ]
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	819 not_ok = [
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	820 "\x7f\xff\xff\xff",
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	821 "\x80\x00\x00\x00",
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	822 "\x81\x00\x00\x00",
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	823 "\x00",
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	824 "\x00\x00\x00\x00\x00",
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	825 ]
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	826 for internal, uni in ok:
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	827 if sys.byteorder == "little":
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	828 internal = "".join(reversed(internal))
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	829 self.assertEquals(uni, internal.decode("unicode_internal"))
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	830 for internal in not_ok:
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	831 if sys.byteorder == "little":
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	832 internal = "".join(reversed(internal))
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	833 self.assertRaises(UnicodeDecodeError, internal.decode,
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	834 "unicode_internal")
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	835
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	836 def test_decode_error_attributes(self):
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	837 if sys.maxunicode > 0xffff:
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	838 try:
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	839 "\x00\x00\x00\x00\x00\x11\x11\x00".decode("unicode_internal")
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	840 except UnicodeDecodeError, ex:
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	841 self.assertEquals("unicode_internal", ex.encoding)
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	842 self.assertEquals("\x00\x00\x00\x00\x00\x11\x11\x00", ex.object)
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	843 self.assertEquals(4, ex.start)
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	844 self.assertEquals(8, ex.end)
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	845 else:
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	846 self.fail()
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	847
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	848 def test_decode_callback(self):
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	849 if sys.maxunicode > 0xffff:
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	850 codecs.register_error("UnicodeInternalTest", codecs.ignore_errors)
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	851 decoder = codecs.getdecoder("unicode_internal")
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	852 ab = u"ab".encode("unicode_internal")
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	853 ignored = decoder("%s\x22\x22\x22\x22%s" % (ab[:4], ab[4:]),
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	854 "UnicodeInternalTest")
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	855 self.assertEquals((u"ab", 12), ignored)
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	856
53855 7dca250343ea Issue 3739: The unicode-internal encoder now reports the number of characters Walter Dörwald <walter@livinglogic.de> parents: 43932 diff changeset	857 def test_encode_length(self):
7dca250343ea Issue 3739: The unicode-internal encoder now reports the number of characters Walter Dörwald <walter@livinglogic.de> parents: 43932 diff changeset	858 # Issue 3739
7dca250343ea Issue 3739: The unicode-internal encoder now reports the number of characters Walter Dörwald <walter@livinglogic.de> parents: 43932 diff changeset	859 encoder = codecs.getencoder("unicode_internal")
7dca250343ea Issue 3739: The unicode-internal encoder now reports the number of characters Walter Dörwald <walter@livinglogic.de> parents: 43932 diff changeset	860 self.assertEquals(encoder(u"a")[1], 1)
7dca250343ea Issue 3739: The unicode-internal encoder now reports the number of characters Walter Dörwald <walter@livinglogic.de> parents: 43932 diff changeset	861 self.assertEquals(encoder(u"\xe9\u0142")[1], 2)
7dca250343ea Issue 3739: The unicode-internal encoder now reports the number of characters Walter Dörwald <walter@livinglogic.de> parents: 43932 diff changeset	862
60339 1cc4d2db5cd6 fix escape_encode to return the correct consumed size Philip Jenvey <pjenvey@underboss.org> parents: 59147 diff changeset	863 encoder = codecs.getencoder("string-escape")
1cc4d2db5cd6 fix escape_encode to return the correct consumed size Philip Jenvey <pjenvey@underboss.org> parents: 59147 diff changeset	864 self.assertEquals(encoder(r'\x00')[1], 4)
1cc4d2db5cd6 fix escape_encode to return the correct consumed size Philip Jenvey <pjenvey@underboss.org> parents: 59147 diff changeset	865
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	866 # From http://www.gnu.org/software/libidn/draft-josefsson-idn-test-vectors.html
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	867 nameprep_tests = [
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	868 # 3.1 Map to nothing.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	869 ('foo\xc2\xad\xcd\x8f\xe1\xa0\x86\xe1\xa0\x8bbar'
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	870 '\xe2\x80\x8b\xe2\x81\xa0baz\xef\xb8\x80\xef\xb8\x88\xef'
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	871 '\xb8\x8f\xef\xbb\xbf',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	872 'foobarbaz'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	873 # 3.2 Case folding ASCII U+0043 U+0041 U+0046 U+0045.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	874 ('CAFE',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	875 'cafe'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	876 # 3.3 Case folding 8bit U+00DF (german sharp s).
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	877 # The original test case is bogus; it says \xc3\xdf
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	878 ('\xc3\x9f',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	879 'ss'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	880 # 3.4 Case folding U+0130 (turkish capital I with dot).
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	881 ('\xc4\xb0',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	882 'i\xcc\x87'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	883 # 3.5 Case folding multibyte U+0143 U+037A.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	884 ('\xc5\x83\xcd\xba',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	885 '\xc5\x84 \xce\xb9'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	886 # 3.6 Case folding U+2121 U+33C6 U+1D7BB.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	887 # XXX: skip this as it fails in UCS-2 mode
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	888 #('\xe2\x84\xa1\xe3\x8f\x86\xf0\x9d\x9e\xbb',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	889 # 'telc\xe2\x88\x95kg\xcf\x83'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	890 (None, None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	891 # 3.7 Normalization of U+006a U+030c U+00A0 U+00AA.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	892 ('j\xcc\x8c\xc2\xa0\xc2\xaa',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	893 '\xc7\xb0 a'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	894 # 3.8 Case folding U+1FB7 and normalization.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	895 ('\xe1\xbe\xb7',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	896 '\xe1\xbe\xb6\xce\xb9'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	897 # 3.9 Self-reverting case folding U+01F0 and normalization.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	898 # The original test case is bogus, it says `\xc7\xf0'
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	899 ('\xc7\xb0',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	900 '\xc7\xb0'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	901 # 3.10 Self-reverting case folding U+0390 and normalization.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	902 ('\xce\x90',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	903 '\xce\x90'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	904 # 3.11 Self-reverting case folding U+03B0 and normalization.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	905 ('\xce\xb0',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	906 '\xce\xb0'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	907 # 3.12 Self-reverting case folding U+1E96 and normalization.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	908 ('\xe1\xba\x96',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	909 '\xe1\xba\x96'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	910 # 3.13 Self-reverting case folding U+1F56 and normalization.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	911 ('\xe1\xbd\x96',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	912 '\xe1\xbd\x96'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	913 # 3.14 ASCII space character U+0020.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	914 (' ',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	915 ' '),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	916 # 3.15 Non-ASCII 8bit space character U+00A0.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	917 ('\xc2\xa0',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	918 ' '),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	919 # 3.16 Non-ASCII multibyte space character U+1680.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	920 ('\xe1\x9a\x80',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	921 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	922 # 3.17 Non-ASCII multibyte space character U+2000.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	923 ('\xe2\x80\x80',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	924 ' '),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	925 # 3.18 Zero Width Space U+200b.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	926 ('\xe2\x80\x8b',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	927 ''),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	928 # 3.19 Non-ASCII multibyte space character U+3000.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	929 ('\xe3\x80\x80',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	930 ' '),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	931 # 3.20 ASCII control characters U+0010 U+007F.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	932 ('\x10\x7f',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	933 '\x10\x7f'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	934 # 3.21 Non-ASCII 8bit control character U+0085.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	935 ('\xc2\x85',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	936 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	937 # 3.22 Non-ASCII multibyte control character U+180E.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	938 ('\xe1\xa0\x8e',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	939 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	940 # 3.23 Zero Width No-Break Space U+FEFF.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	941 ('\xef\xbb\xbf',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	942 ''),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	943 # 3.24 Non-ASCII control character U+1D175.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	944 ('\xf0\x9d\x85\xb5',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	945 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	946 # 3.25 Plane 0 private use character U+F123.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	947 ('\xef\x84\xa3',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	948 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	949 # 3.26 Plane 15 private use character U+F1234.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	950 ('\xf3\xb1\x88\xb4',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	951 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	952 # 3.27 Plane 16 private use character U+10F234.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	953 ('\xf4\x8f\x88\xb4',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	954 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	955 # 3.28 Non-character code point U+8FFFE.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	956 ('\xf2\x8f\xbf\xbe',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	957 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	958 # 3.29 Non-character code point U+10FFFF.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	959 ('\xf4\x8f\xbf\xbf',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	960 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	961 # 3.30 Surrogate code U+DF42.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	962 ('\xed\xbd\x82',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	963 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	964 # 3.31 Non-plain text character U+FFFD.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	965 ('\xef\xbf\xbd',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	966 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	967 # 3.32 Ideographic description character U+2FF5.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	968 ('\xe2\xbf\xb5',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	969 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	970 # 3.33 Display property character U+0341.
28774 c9bee11eaa8b Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 28678 diff changeset	971 ('\xcd\x81',
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	972 '\xcc\x81'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	973 # 3.34 Left-to-right mark U+200E.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	974 ('\xe2\x80\x8e',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	975 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	976 # 3.35 Deprecated U+202A.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	977 ('\xe2\x80\xaa',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	978 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	979 # 3.36 Language tagging character U+E0001.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	980 ('\xf3\xa0\x80\x81',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	981 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	982 # 3.37 Language tagging character U+E0042.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	983 ('\xf3\xa0\x81\x82',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	984 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	985 # 3.38 Bidi: RandALCat character U+05BE and LCat characters.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	986 ('foo\xd6\xbebar',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	987 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	988 # 3.39 Bidi: RandALCat character U+FD50 and LCat characters.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	989 ('foo\xef\xb5\x90bar',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	990 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	991 # 3.40 Bidi: RandALCat character U+FB38 and LCat characters.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	992 ('foo\xef\xb9\xb6bar',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	993 'foo \xd9\x8ebar'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	994 # 3.41 Bidi: RandALCat without trailing RandALCat U+0627 U+0031.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	995 ('\xd8\xa71',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	996 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	997 # 3.42 Bidi: RandALCat character U+0627 U+0031 U+0628.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	998 ('\xd8\xa71\xd8\xa8',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	999 '\xd8\xa71\xd8\xa8'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1000 # 3.43 Unassigned code point U+E0002.
28678 31cb8e0260db Skip nameprep test 3.43, as we do allow unassigned characters. The test Martin v. Löwis <martin@v.loewis.de> parents: 28671 diff changeset	1001 # Skip this test as we allow unassigned
31cb8e0260db Skip nameprep test 3.43, as we do allow unassigned characters. The test Martin v. Löwis <martin@v.loewis.de> parents: 28671 diff changeset	1002 #('\xf3\xa0\x80\x82',
31cb8e0260db Skip nameprep test 3.43, as we do allow unassigned characters. The test Martin v. Löwis <martin@v.loewis.de> parents: 28671 diff changeset	1003 # None),
31cb8e0260db Skip nameprep test 3.43, as we do allow unassigned characters. The test Martin v. Löwis <martin@v.loewis.de> parents: 28671 diff changeset	1004 (None, None),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1005 # 3.44 Larger test (shrinking).
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1006 # Original test case reads \xc3\xdf
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1007 ('X\xc2\xad\xc3\x9f\xc4\xb0\xe2\x84\xa1j\xcc\x8c\xc2\xa0\xc2'
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1008 '\xaa\xce\xb0\xe2\x80\x80',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1009 'xssi\xcc\x87tel\xc7\xb0 a\xce\xb0 '),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1010 # 3.45 Larger test (expanding).
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1011 # Original test case reads \xc3\x9f
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1012 ('X\xc3\x9f\xe3\x8c\x96\xc4\xb0\xe2\x84\xa1\xe2\x92\x9f\xe3\x8c'
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1013 '\x80',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1014 'xss\xe3\x82\xad\xe3\x83\xad\xe3\x83\xa1\xe3\x83\xbc\xe3'
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1015 '\x83\x88\xe3\x83\xabi\xcc\x87tel\x28d\x29\xe3\x82'
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1016 '\xa2\xe3\x83\x91\xe3\x83\xbc\xe3\x83\x88')
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1017 ]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1018
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1019
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1020 class NameprepTest(unittest.TestCase):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1021 def test_nameprep(self):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1022 from encodings.idna import nameprep
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1023 for pos, (orig, prepped) in enumerate(nameprep_tests):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1024 if orig is None:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1025 # Skipped
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1026 continue
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1027 # The Unicode strings are given in UTF-8
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1028 orig = unicode(orig, "utf-8")
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1029 if prepped is None:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1030 # Input contains prohibited characters
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1031 self.assertRaises(UnicodeError, nameprep, orig)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1032 else:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1033 prepped = unicode(prepped, "utf-8")
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1034 try:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1035 self.assertEquals(nameprep(orig), prepped)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1036 except Exception,e:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1037 raise test_support.TestFailed("Test 3.%d: %s" % (pos+1, str(e)))
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1038
37719 cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1039 class IDNACodecTest(unittest.TestCase):
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1040 def test_builtin_decode(self):
31795 bf37429f6d9f Add test case for unicode(somestring, "idna"). Martin v. Löwis <martin@v.loewis.de> parents: 28917 diff changeset	1041 self.assertEquals(unicode("python.org", "idna"), u"python.org")
37719 cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1042 self.assertEquals(unicode("python.org.", "idna"), u"python.org.")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1043 self.assertEquals(unicode("xn--pythn-mua.org", "idna"), u"pyth\xf6n.org")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1044 self.assertEquals(unicode("xn--pythn-mua.org.", "idna"), u"pyth\xf6n.org.")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1045
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1046 def test_builtin_encode(self):
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1047 self.assertEquals(u"python.org".encode("idna"), "python.org")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1048 self.assertEquals("python.org.".encode("idna"), "python.org.")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1049 self.assertEquals(u"pyth\xf6n.org".encode("idna"), "xn--pythn-mua.org")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1050 self.assertEquals(u"pyth\xf6n.org.".encode("idna"), "xn--pythn-mua.org.")
31795 bf37429f6d9f Add test case for unicode(somestring, "idna"). Martin v. Löwis <martin@v.loewis.de> parents: 28917 diff changeset	1051
35435 9d343bc81863 Make IDNA return an empty string when the input is empty. Fixes #1163178. Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	1052 def test_stream(self):
9d343bc81863 Make IDNA return an empty string when the input is empty. Fixes #1163178. Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	1053 import StringIO
9d343bc81863 Make IDNA return an empty string when the input is empty. Fixes #1163178. Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	1054 r = codecs.getreader("idna")(StringIO.StringIO("abc"))
9d343bc81863 Make IDNA return an empty string when the input is empty. Fixes #1163178. Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	1055 r.read(3)
9d343bc81863 Make IDNA return an empty string when the input is empty. Fixes #1163178. Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	1056 self.assertEquals(r.read(), u"")
9d343bc81863 Make IDNA return an empty string when the input is empty. Fixes #1163178. Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	1057
37719 cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1058 def test_incremental_decode(self):
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1059 self.assertEquals(
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1060 "".join(codecs.iterdecode("python.org", "idna")),
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1061 u"python.org"
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1062 )
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1063 self.assertEquals(
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1064 "".join(codecs.iterdecode("python.org.", "idna")),
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1065 u"python.org."
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1066 )
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1067 self.assertEquals(
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1068 "".join(codecs.iterdecode("xn--pythn-mua.org.", "idna")),
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1069 u"pyth\xf6n.org."
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1070 )
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1071 self.assertEquals(
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1072 "".join(codecs.iterdecode("xn--pythn-mua.org.", "idna")),
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1073 u"pyth\xf6n.org."
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1074 )
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1075
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1076 decoder = codecs.getincrementaldecoder("idna")()
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1077 self.assertEquals(decoder.decode("xn--xam", ), u"")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1078 self.assertEquals(decoder.decode("ple-9ta.o", ), u"\xe4xample.")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1079 self.assertEquals(decoder.decode(u"rg"), u"")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1080 self.assertEquals(decoder.decode(u"", True), u"org")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1081
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1082 decoder.reset()
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1083 self.assertEquals(decoder.decode("xn--xam", ), u"")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1084 self.assertEquals(decoder.decode("ple-9ta.o", ), u"\xe4xample.")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1085 self.assertEquals(decoder.decode("rg."), u"org.")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1086 self.assertEquals(decoder.decode("", True), u"")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1087
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1088 def test_incremental_encode(self):
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1089 self.assertEquals(
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1090 "".join(codecs.iterencode(u"python.org", "idna")),
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1091 "python.org"
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1092 )
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1093 self.assertEquals(
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1094 "".join(codecs.iterencode(u"python.org.", "idna")),
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1095 "python.org."
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1096 )
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1097 self.assertEquals(
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1098 "".join(codecs.iterencode(u"pyth\xf6n.org.", "idna")),
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1099 "xn--pythn-mua.org."
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1100 )
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1101 self.assertEquals(
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1102 "".join(codecs.iterencode(u"pyth\xf6n.org.", "idna")),
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1103 "xn--pythn-mua.org."
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1104 )
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1105
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1106 encoder = codecs.getincrementalencoder("idna")()
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1107 self.assertEquals(encoder.encode(u"\xe4x"), "")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1108 self.assertEquals(encoder.encode(u"ample.org"), "xn--xample-9ta.")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1109 self.assertEquals(encoder.encode(u"", True), "org")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1110
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1111 encoder.reset()
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1112 self.assertEquals(encoder.encode(u"\xe4x"), "")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1113 self.assertEquals(encoder.encode(u"ample.org."), "xn--xample-9ta.org.")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1114 self.assertEquals(encoder.encode(u"", True), "")
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1115
32572 8ea2cb1ec598 Add generic codecs.encode() and .decode() APIs that don't impose Marc-André Lemburg <mal@egenix.com> parents: 32538 diff changeset	1116 class CodecsModuleTest(unittest.TestCase):
8ea2cb1ec598 Add generic codecs.encode() and .decode() APIs that don't impose Marc-André Lemburg <mal@egenix.com> parents: 32538 diff changeset	1117
8ea2cb1ec598 Add generic codecs.encode() and .decode() APIs that don't impose Marc-André Lemburg <mal@egenix.com> parents: 32538 diff changeset	1118 def test_decode(self):
8ea2cb1ec598 Add generic codecs.encode() and .decode() APIs that don't impose Marc-André Lemburg <mal@egenix.com> parents: 32538 diff changeset	1119 self.assertEquals(codecs.decode('\xe4\xf6\xfc', 'latin-1'),
8ea2cb1ec598 Add generic codecs.encode() and .decode() APIs that don't impose Marc-André Lemburg <mal@egenix.com> parents: 32538 diff changeset	1120 u'\xe4\xf6\xfc')
33855 1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1121 self.assertRaises(TypeError, codecs.decode)
1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1122 self.assertEquals(codecs.decode('abc'), u'abc')
1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1123 self.assertRaises(UnicodeDecodeError, codecs.decode, '\xff', 'ascii')
1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1124
32572 8ea2cb1ec598 Add generic codecs.encode() and .decode() APIs that don't impose Marc-André Lemburg <mal@egenix.com> parents: 32538 diff changeset	1125 def test_encode(self):
8ea2cb1ec598 Add generic codecs.encode() and .decode() APIs that don't impose Marc-André Lemburg <mal@egenix.com> parents: 32538 diff changeset	1126 self.assertEquals(codecs.encode(u'\xe4\xf6\xfc', 'latin-1'),
8ea2cb1ec598 Add generic codecs.encode() and .decode() APIs that don't impose Marc-André Lemburg <mal@egenix.com> parents: 32538 diff changeset	1127 '\xe4\xf6\xfc')
33855 1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1128 self.assertRaises(TypeError, codecs.encode)
35889 e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1129 self.assertRaises(LookupError, codecs.encode, "foo", "__spam__")
33855 1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1130 self.assertEquals(codecs.encode(u'abc'), 'abc')
1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1131 self.assertRaises(UnicodeEncodeError, codecs.encode, u'\xffff', 'ascii')
1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1132
1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1133 def test_register(self):
1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1134 self.assertRaises(TypeError, codecs.register)
35889 e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1135 self.assertRaises(TypeError, codecs.register, 42)
33855 1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1136
1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1137 def test_lookup(self):
1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1138 self.assertRaises(TypeError, codecs.lookup)
1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1139 self.assertRaises(LookupError, codecs.lookup, "__spam__")
35889 e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1140 self.assertRaises(LookupError, codecs.lookup, " ")
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1141
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1142 def test_getencoder(self):
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1143 self.assertRaises(TypeError, codecs.getencoder)
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1144 self.assertRaises(LookupError, codecs.getencoder, "__spam__")
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1145
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1146 def test_getdecoder(self):
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1147 self.assertRaises(TypeError, codecs.getdecoder)
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1148 self.assertRaises(LookupError, codecs.getdecoder, "__spam__")
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1149
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1150 def test_getreader(self):
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1151 self.assertRaises(TypeError, codecs.getreader)
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1152 self.assertRaises(LookupError, codecs.getreader, "__spam__")
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1153
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1154 def test_getwriter(self):
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1155 self.assertRaises(TypeError, codecs.getwriter)
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1156 self.assertRaises(LookupError, codecs.getwriter, "__spam__")
32572 8ea2cb1ec598 Add generic codecs.encode() and .decode() APIs that don't impose Marc-André Lemburg <mal@egenix.com> parents: 32538 diff changeset	1157
33782 a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1158 class StreamReaderTest(unittest.TestCase):
a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1159
a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1160 def setUp(self):
a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1161 self.reader = codecs.getreader('utf-8')
a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1162 self.stream = StringIO.StringIO('\xed\x95\x9c\n\xea\xb8\x80')
a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1163
a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1164 def test_readlines(self):
a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1165 f = self.reader(self.stream)
a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1166 self.assertEquals(f.readlines(), [u'\ud55c\n', u'\uae00'])
a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1167
40239 7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1168 class EncodedFileTest(unittest.TestCase):
40284 ff678669aa6e Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 40274 diff changeset	1169
40239 7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1170 def test_basic(self):
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1171 f = StringIO.StringIO('\xed\x95\x9c\n\xea\xb8\x80')
40249 9663100c95ca Fix the new EncodedFile test to work with big endian platforms. Georg Brandl <georg@python.org> parents: 40239 diff changeset	1172 ef = codecs.EncodedFile(f, 'utf-16-le', 'utf-8')
9663100c95ca Fix the new EncodedFile test to work with big endian platforms. Georg Brandl <georg@python.org> parents: 40239 diff changeset	1173 self.assertEquals(ef.read(), '\\\xd5\n\x00\x00\xae')
40239 7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1174
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1175 f = StringIO.StringIO()
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1176 ef = codecs.EncodedFile(f, 'utf-8', 'latin1')
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1177 ef.write('\xc3\xbc')
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1178 self.assertEquals(f.getvalue(), '\xfc')
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1179
35269 2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1180 class Str2StrTest(unittest.TestCase):
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1181
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1182 def test_read(self):
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1183 sin = "\x80".encode("base64_codec")
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1184 reader = codecs.getreader("base64_codec")(StringIO.StringIO(sin))
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1185 sout = reader.read()
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1186 self.assertEqual(sout, "\x80")
58472 6ffcdafa0118 use assert[Not]IsInstance where appropriate Ezio Melotti <ezio.melotti@gmail.com> parents: 56033 diff changeset	1187 self.assertIsInstance(sout, str)
35269 2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1188
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1189 def test_readline(self):
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1190 sin = "\x80".encode("base64_codec")
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1191 reader = codecs.getreader("base64_codec")(StringIO.StringIO(sin))
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1192 sout = reader.readline()
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1193 self.assertEqual(sout, "\x80")
58472 6ffcdafa0118 use assert[Not]IsInstance where appropriate Ezio Melotti <ezio.melotti@gmail.com> parents: 56033 diff changeset	1194 self.assertIsInstance(sout, str)
35269 2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1195
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1196 all_unicode_encodings = [
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1197 "ascii",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1198 "base64_codec",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1199 "big5",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1200 "big5hkscs",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1201 "charmap",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1202 "cp037",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1203 "cp1006",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1204 "cp1026",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1205 "cp1140",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1206 "cp1250",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1207 "cp1251",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1208 "cp1252",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1209 "cp1253",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1210 "cp1254",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1211 "cp1255",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1212 "cp1256",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1213 "cp1257",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1214 "cp1258",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1215 "cp424",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1216 "cp437",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1217 "cp500",
61884 c56fb63c648f #8016: add the CP858 codec (approved by Benjamin). (Also add CP720 to the tests, it was missing there.) Georg Brandl <georg@python.org> parents: 61859 diff changeset	1218 "cp720",
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1219 "cp737",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1220 "cp775",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1221 "cp850",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1222 "cp852",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1223 "cp855",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1224 "cp856",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1225 "cp857",
61884 c56fb63c648f #8016: add the CP858 codec (approved by Benjamin). (Also add CP720 to the tests, it was missing there.) Georg Brandl <georg@python.org> parents: 61859 diff changeset	1226 "cp858",
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1227 "cp860",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1228 "cp861",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1229 "cp862",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1230 "cp863",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1231 "cp864",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1232 "cp865",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1233 "cp866",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1234 "cp869",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1235 "cp874",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1236 "cp875",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1237 "cp932",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1238 "cp949",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1239 "cp950",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1240 "euc_jis_2004",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1241 "euc_jisx0213",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1242 "euc_jp",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1243 "euc_kr",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1244 "gb18030",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1245 "gb2312",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1246 "gbk",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1247 "hex_codec",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1248 "hp_roman8",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1249 "hz",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1250 "idna",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1251 "iso2022_jp",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1252 "iso2022_jp_1",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1253 "iso2022_jp_2",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1254 "iso2022_jp_2004",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1255 "iso2022_jp_3",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1256 "iso2022_jp_ext",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1257 "iso2022_kr",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1258 "iso8859_1",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1259 "iso8859_10",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1260 "iso8859_11",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1261 "iso8859_13",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1262 "iso8859_14",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1263 "iso8859_15",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1264 "iso8859_16",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1265 "iso8859_2",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1266 "iso8859_3",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1267 "iso8859_4",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1268 "iso8859_5",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1269 "iso8859_6",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1270 "iso8859_7",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1271 "iso8859_8",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1272 "iso8859_9",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1273 "johab",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1274 "koi8_r",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1275 "koi8_u",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1276 "latin_1",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1277 "mac_cyrillic",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1278 "mac_greek",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1279 "mac_iceland",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1280 "mac_latin2",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1281 "mac_roman",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1282 "mac_turkish",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1283 "palmos",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1284 "ptcp154",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1285 "punycode",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1286 "raw_unicode_escape",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1287 "rot_13",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1288 "shift_jis",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1289 "shift_jis_2004",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1290 "shift_jisx0213",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1291 "tis_620",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1292 "unicode_escape",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1293 "unicode_internal",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1294 "utf_16",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1295 "utf_16_be",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1296 "utf_16_le",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1297 "utf_7",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1298 "utf_8",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1299 ]
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1300
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1301 if hasattr(codecs, "mbcs_encode"):
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1302 all_unicode_encodings.append("mbcs")
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1303
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1304 # The following encodings work only with str, not unicode
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1305 all_string_encodings = [
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1306 "quopri_codec",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1307 "string_escape",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1308 "uu_codec",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1309 ]
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1310
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1311 # The following encoding is not tested, because it's not supposed
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1312 # to work:
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1313 # "undefined"
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1314
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1315 # The following encodings don't work in stateful mode
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1316 broken_unicode_with_streams = [
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1317 "base64_codec",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1318 "hex_codec",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1319 "punycode",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1320 "unicode_internal"
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1321 ]
40251 e16800ac1eea Bug #1586613: fix zlib and bz2 codecs' incremental en/decoders. Georg Brandl <georg@python.org> parents: 40249 diff changeset	1322 broken_incremental_coders = broken_unicode_with_streams[:]
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1323
40271 c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1324 # The following encodings only support "strict" mode
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1325 only_strict_mode = [
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1326 "idna",
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1327 "zlib_codec",
40274 bc8ffc3e3cdb I'm assuming this is correct, it fixes the tests so they pass again Neal Norwitz <nnorwitz@gmail.com> parents: 40271 diff changeset	1328 "bz2_codec",
40271 c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1329 ]
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1330
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1331 try:
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1332 import bz2
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1333 except ImportError:
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1334 pass
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1335 else:
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1336 all_unicode_encodings.append("bz2_codec")
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1337 broken_unicode_with_streams.append("bz2_codec")
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1338
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1339 try:
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1340 import zlib
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1341 except ImportError:
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1342 pass
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1343 else:
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1344 all_unicode_encodings.append("zlib_codec")
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1345 broken_unicode_with_streams.append("zlib_codec")
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1346
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1347 class BasicUnicodeTest(unittest.TestCase):
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1348 def test_basics(self):
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1349 s = u"abc123" # all codecs should be able to encode these
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1350 for encoding in all_unicode_encodings:
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1351 name = codecs.lookup(encoding).name
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1352 if encoding.endswith("_codec"):
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1353 name += "_codec"
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1354 elif encoding == "latin_1":
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1355 name = "latin_1"
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1356 self.assertEqual(encoding.replace("_", "-"), name.replace("_", "-"))
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1357 (bytes, size) = codecs.getencoder(encoding)(s)
53855 7dca250343ea Issue 3739: The unicode-internal encoder now reports the number of characters Walter Dörwald <walter@livinglogic.de> parents: 43932 diff changeset	1358 self.assertEqual(size, len(s), "%r != %r (encoding=%r)" % (size, len(s), encoding))
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1359 (chars, size) = codecs.getdecoder(encoding)(bytes)
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1360 self.assertEqual(chars, s, "%r != %r (encoding=%r)" % (chars, s, encoding))
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1361
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1362 if encoding not in broken_unicode_with_streams:
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1363 # check stream reader/writer
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1364 q = Queue()
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1365 writer = codecs.getwriter(encoding)(q)
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1366 encodedresult = ""
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1367 for c in s:
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1368 writer.write(c)
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1369 encodedresult += q.read()
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1370 q = Queue()
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1371 reader = codecs.getreader(encoding)(q)
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1372 decodedresult = u""
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1373 for c in encodedresult:
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1374 q.write(c)
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1375 decodedresult += reader.read()
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1376 self.assertEqual(decodedresult, s, "%r != %r (encoding=%r)" % (decodedresult, s, encoding))
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1377
40251 e16800ac1eea Bug #1586613: fix zlib and bz2 codecs' incremental en/decoders. Georg Brandl <georg@python.org> parents: 40249 diff changeset	1378 if encoding not in broken_incremental_coders:
37077 38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	1379 # check incremental decoder/encoder (fetched via the Python
38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	1380 # and C API) and iterencode()/iterdecode()
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1381 try:
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1382 encoder = codecs.getincrementalencoder(encoding)()
37077 38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	1383 cencoder = _testcapi.codec_incrementalencoder(encoding)
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1384 except LookupError: # no IncrementalEncoder
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1385 pass
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1386 else:
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1387 # check incremental decoder/encoder
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1388 encodedresult = ""
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1389 for c in s:
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1390 encodedresult += encoder.encode(c)
37708 230fff6e0618 Call encode()/decode() with final==True as the last call in the Walter Dörwald <walter@livinglogic.de> parents: 37077 diff changeset	1391 encodedresult += encoder.encode(u"", True)
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1392 decoder = codecs.getincrementaldecoder(encoding)()
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1393 decodedresult = u""
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1394 for c in encodedresult:
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1395 decodedresult += decoder.decode(c)
37708 230fff6e0618 Call encode()/decode() with final==True as the last call in the Walter Dörwald <walter@livinglogic.de> parents: 37077 diff changeset	1396 decodedresult += decoder.decode("", True)
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1397 self.assertEqual(decodedresult, s, "%r != %r (encoding=%r)" % (decodedresult, s, encoding))
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1398
37077 38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	1399 # check C API
38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	1400 encodedresult = ""
38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	1401 for c in s:
38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	1402 encodedresult += cencoder.encode(c)
37708 230fff6e0618 Call encode()/decode() with final==True as the last call in the Walter Dörwald <walter@livinglogic.de> parents: 37077 diff changeset	1403 encodedresult += cencoder.encode(u"", True)
37077 38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	1404 cdecoder = _testcapi.codec_incrementaldecoder(encoding)
38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	1405 decodedresult = u""
38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	1406 for c in encodedresult:
38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	1407 decodedresult += cdecoder.decode(c)
37708 230fff6e0618 Call encode()/decode() with final==True as the last call in the Walter Dörwald <walter@livinglogic.de> parents: 37077 diff changeset	1408 decodedresult += cdecoder.decode("", True)
37077 38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	1409 self.assertEqual(decodedresult, s, "%r != %r (encoding=%r)" % (decodedresult, s, encoding))
38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	1410
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1411 # check iterencode()/iterdecode()
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1412 result = u"".join(codecs.iterdecode(codecs.iterencode(s, encoding), encoding))
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1413 self.assertEqual(result, s, "%r != %r (encoding=%r)" % (result, s, encoding))
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1414
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1415 # check iterencode()/iterdecode() with empty string
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1416 result = u"".join(codecs.iterdecode(codecs.iterencode(u"", encoding), encoding))
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1417 self.assertEqual(result, u"")
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1418
40271 c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1419 if encoding not in only_strict_mode:
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1420 # check incremental decoder/encoder with errors argument
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1421 try:
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1422 encoder = codecs.getincrementalencoder(encoding)("ignore")
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1423 cencoder = _testcapi.codec_incrementalencoder(encoding, "ignore")
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1424 except LookupError: # no IncrementalEncoder
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1425 pass
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1426 else:
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1427 encodedresult = "".join(encoder.encode(c) for c in s)
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1428 decoder = codecs.getincrementaldecoder(encoding)("ignore")
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1429 decodedresult = u"".join(decoder.decode(c) for c in encodedresult)
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1430 self.assertEqual(decodedresult, s, "%r != %r (encoding=%r)" % (decodedresult, s, encoding))
40284 ff678669aa6e Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 40274 diff changeset	1431
40271 c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1432 encodedresult = "".join(cencoder.encode(c) for c in s)
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1433 cdecoder = _testcapi.codec_incrementaldecoder(encoding, "ignore")
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1434 decodedresult = u"".join(cdecoder.decode(c) for c in encodedresult)
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1435 self.assertEqual(decodedresult, s, "%r != %r (encoding=%r)" % (decodedresult, s, encoding))
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1436
34750 679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1437 def test_seek(self):
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1438 # all codecs should be able to encode these
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1439 s = u"%s\n%s\n" % (100u"abc123", 100u"def456")
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1440 for encoding in all_unicode_encodings:
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1441 if encoding == "idna": # FIXME: See SF bug #1163178
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1442 continue
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1443 if encoding in broken_unicode_with_streams:
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1444 continue
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1445 reader = codecs.getreader(encoding)(StringIO.StringIO(s.encode(encoding)))
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1446 for t in xrange(5):
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1447 # Test that calling seek resets the internal codec state and buffers
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1448 reader.seek(0, 0)
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1449 line = reader.readline()
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1450 self.assertEqual(s[:len(line)], line)
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1451
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1452 def test_bad_decode_args(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1453 for encoding in all_unicode_encodings:
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1454 decoder = codecs.getdecoder(encoding)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1455 self.assertRaises(TypeError, decoder)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1456 if encoding not in ("idna", "punycode"):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1457 self.assertRaises(TypeError, decoder, 42)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1458
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1459 def test_bad_encode_args(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1460 for encoding in all_unicode_encodings:
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1461 encoder = codecs.getencoder(encoding)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1462 self.assertRaises(TypeError, encoder)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1463
38787 70b166ca1c49 Verify the crash due to EncodingMap not initialized does not return Neal Norwitz <nnorwitz@gmail.com> parents: 37719 diff changeset	1464 def test_encoding_map_type_initialized(self):
70b166ca1c49 Verify the crash due to EncodingMap not initialized does not return Neal Norwitz <nnorwitz@gmail.com> parents: 37719 diff changeset	1465 from encodings import cp1140
70b166ca1c49 Verify the crash due to EncodingMap not initialized does not return Neal Norwitz <nnorwitz@gmail.com> parents: 37719 diff changeset	1466 # This used to crash, we are only verifying there's no crash.
70b166ca1c49 Verify the crash due to EncodingMap not initialized does not return Neal Norwitz <nnorwitz@gmail.com> parents: 37719 diff changeset	1467 table_type = type(cp1140.encoding_table)
70b166ca1c49 Verify the crash due to EncodingMap not initialized does not return Neal Norwitz <nnorwitz@gmail.com> parents: 37719 diff changeset	1468 self.assertEqual(table_type, table_type)
70b166ca1c49 Verify the crash due to EncodingMap not initialized does not return Neal Norwitz <nnorwitz@gmail.com> parents: 37719 diff changeset	1469
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1470 class BasicStrTest(unittest.TestCase):
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1471 def test_basics(self):
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1472 s = "abc123"
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1473 for encoding in all_string_encodings:
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1474 (bytes, size) = codecs.getencoder(encoding)(s)
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1475 self.assertEqual(size, len(s))
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1476 (chars, size) = codecs.getdecoder(encoding)(bytes)
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1477 self.assertEqual(chars, s, "%r != %r (encoding=%r)" % (chars, s, encoding))
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1478
35702 22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1479 class CharmapTest(unittest.TestCase):
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1480 def test_decode_with_string_map(self):
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1481 self.assertEquals(
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1482 codecs.charmap_decode("\x00\x01\x02", "strict", u"abc"),
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1483 (u"abc", 3)
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1484 )
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1485
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1486 self.assertEquals(
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1487 codecs.charmap_decode("\x00\x01\x02", "replace", u"ab"),
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1488 (u"ab\ufffd", 3)
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1489 )
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1490
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1491 self.assertEquals(
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1492 codecs.charmap_decode("\x00\x01\x02", "replace", u"ab\ufffe"),
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1493 (u"ab\ufffd", 3)
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1494 )
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1495
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1496 self.assertEquals(
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1497 codecs.charmap_decode("\x00\x01\x02", "ignore", u"ab"),
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1498 (u"ab", 3)
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1499 )
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1500
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1501 self.assertEquals(
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1502 codecs.charmap_decode("\x00\x01\x02", "ignore", u"ab\ufffe"),
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1503 (u"ab", 3)
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1504 )
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1505
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1506 allbytes = "".join(chr(i) for i in xrange(256))
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1507 self.assertEquals(
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1508 codecs.charmap_decode(allbytes, "ignore", u""),
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1509 (u"", len(allbytes))
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1510 )
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1511
40239 7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1512 class WithStmtTest(unittest.TestCase):
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1513 def test_encodedfile(self):
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1514 f = StringIO.StringIO("\xc3\xbc")
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1515 with codecs.EncodedFile(f, "latin-1", "utf-8") as ef:
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1516 self.assertEquals(ef.read(), "\xfc")
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1517
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1518 def test_streamreaderwriter(self):
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1519 f = StringIO.StringIO("\xc3\xbc")
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1520 info = codecs.lookup("utf-8")
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1521 with codecs.StreamReaderWriter(f, info.streamreader,
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1522 info.streamwriter, 'strict') as srw:
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1523 self.assertEquals(srw.read(), u"\xfc")
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1524
35702 22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1525
61847 99e9ed214aad Issue #6268: Fix seek() method of codecs.open(), don't read the BOM twice Victor Stinner <victor.stinner@haypocalc.com> parents: 60339 diff changeset	1526 class BomTest(unittest.TestCase):
99e9ed214aad Issue #6268: Fix seek() method of codecs.open(), don't read the BOM twice Victor Stinner <victor.stinner@haypocalc.com> parents: 60339 diff changeset	1527 def test_seek0(self):
61859 bf82c1d04e06 Issue #6268: More bugfixes about BOM, UTF-16 and UTF-32 Victor Stinner <victor.stinner@haypocalc.com> parents: 61847 diff changeset	1528 data = u"1234567890"
61847 99e9ed214aad Issue #6268: Fix seek() method of codecs.open(), don't read the BOM twice Victor Stinner <victor.stinner@haypocalc.com> parents: 60339 diff changeset	1529 tests = ("utf-16",
99e9ed214aad Issue #6268: Fix seek() method of codecs.open(), don't read the BOM twice Victor Stinner <victor.stinner@haypocalc.com> parents: 60339 diff changeset	1530 "utf-16-le",
99e9ed214aad Issue #6268: Fix seek() method of codecs.open(), don't read the BOM twice Victor Stinner <victor.stinner@haypocalc.com> parents: 60339 diff changeset	1531 "utf-16-be",
99e9ed214aad Issue #6268: Fix seek() method of codecs.open(), don't read the BOM twice Victor Stinner <victor.stinner@haypocalc.com> parents: 60339 diff changeset	1532 "utf-32",
99e9ed214aad Issue #6268: Fix seek() method of codecs.open(), don't read the BOM twice Victor Stinner <victor.stinner@haypocalc.com> parents: 60339 diff changeset	1533 "utf-32-le",
99e9ed214aad Issue #6268: Fix seek() method of codecs.open(), don't read the BOM twice Victor Stinner <victor.stinner@haypocalc.com> parents: 60339 diff changeset	1534 "utf-32-be")
99e9ed214aad Issue #6268: Fix seek() method of codecs.open(), don't read the BOM twice Victor Stinner <victor.stinner@haypocalc.com> parents: 60339 diff changeset	1535 for encoding in tests:
61859 bf82c1d04e06 Issue #6268: More bugfixes about BOM, UTF-16 and UTF-32 Victor Stinner <victor.stinner@haypocalc.com> parents: 61847 diff changeset	1536 # Check if the BOM is written only once
bf82c1d04e06 Issue #6268: More bugfixes about BOM, UTF-16 and UTF-32 Victor Stinner <victor.stinner@haypocalc.com> parents: 61847 diff changeset	1537 with codecs.open(test_support.TESTFN, 'w+', encoding=encoding) as f:
61847 99e9ed214aad Issue #6268: Fix seek() method of codecs.open(), don't read the BOM twice Victor Stinner <victor.stinner@haypocalc.com> parents: 60339 diff changeset	1538 f.write(data)
99e9ed214aad Issue #6268: Fix seek() method of codecs.open(), don't read the BOM twice Victor Stinner <victor.stinner@haypocalc.com> parents: 60339 diff changeset	1539 f.write(data)
99e9ed214aad Issue #6268: Fix seek() method of codecs.open(), don't read the BOM twice Victor Stinner <victor.stinner@haypocalc.com> parents: 60339 diff changeset	1540 f.seek(0)
99e9ed214aad Issue #6268: Fix seek() method of codecs.open(), don't read the BOM twice Victor Stinner <victor.stinner@haypocalc.com> parents: 60339 diff changeset	1541 self.assertEquals(f.read(), data * 2)
99e9ed214aad Issue #6268: Fix seek() method of codecs.open(), don't read the BOM twice Victor Stinner <victor.stinner@haypocalc.com> parents: 60339 diff changeset	1542 f.seek(0)
99e9ed214aad Issue #6268: Fix seek() method of codecs.open(), don't read the BOM twice Victor Stinner <victor.stinner@haypocalc.com> parents: 60339 diff changeset	1543 self.assertEquals(f.read(), data * 2)
99e9ed214aad Issue #6268: Fix seek() method of codecs.open(), don't read the BOM twice Victor Stinner <victor.stinner@haypocalc.com> parents: 60339 diff changeset	1544
61859 bf82c1d04e06 Issue #6268: More bugfixes about BOM, UTF-16 and UTF-32 Victor Stinner <victor.stinner@haypocalc.com> parents: 61847 diff changeset	1545 # Check that the BOM is written after a seek(0)
bf82c1d04e06 Issue #6268: More bugfixes about BOM, UTF-16 and UTF-32 Victor Stinner <victor.stinner@haypocalc.com> parents: 61847 diff changeset	1546 with codecs.open(test_support.TESTFN, 'w+', encoding=encoding) as f:
bf82c1d04e06 Issue #6268: More bugfixes about BOM, UTF-16 and UTF-32 Victor Stinner <victor.stinner@haypocalc.com> parents: 61847 diff changeset	1547 f.write(data[0])
bf82c1d04e06 Issue #6268: More bugfixes about BOM, UTF-16 and UTF-32 Victor Stinner <victor.stinner@haypocalc.com> parents: 61847 diff changeset	1548 self.assertNotEquals(f.tell(), 0)
bf82c1d04e06 Issue #6268: More bugfixes about BOM, UTF-16 and UTF-32 Victor Stinner <victor.stinner@haypocalc.com> parents: 61847 diff changeset	1549 f.seek(0)
bf82c1d04e06 Issue #6268: More bugfixes about BOM, UTF-16 and UTF-32 Victor Stinner <victor.stinner@haypocalc.com> parents: 61847 diff changeset	1550 f.write(data)
bf82c1d04e06 Issue #6268: More bugfixes about BOM, UTF-16 and UTF-32 Victor Stinner <victor.stinner@haypocalc.com> parents: 61847 diff changeset	1551 f.seek(0)
bf82c1d04e06 Issue #6268: More bugfixes about BOM, UTF-16 and UTF-32 Victor Stinner <victor.stinner@haypocalc.com> parents: 61847 diff changeset	1552 self.assertEquals(f.read(), data)
bf82c1d04e06 Issue #6268: More bugfixes about BOM, UTF-16 and UTF-32 Victor Stinner <victor.stinner@haypocalc.com> parents: 61847 diff changeset	1553
bf82c1d04e06 Issue #6268: More bugfixes about BOM, UTF-16 and UTF-32 Victor Stinner <victor.stinner@haypocalc.com> parents: 61847 diff changeset	1554 # (StreamWriter) Check that the BOM is written after a seek(0)
bf82c1d04e06 Issue #6268: More bugfixes about BOM, UTF-16 and UTF-32 Victor Stinner <victor.stinner@haypocalc.com> parents: 61847 diff changeset	1555 with codecs.open(test_support.TESTFN, 'w+', encoding=encoding) as f:
bf82c1d04e06 Issue #6268: More bugfixes about BOM, UTF-16 and UTF-32 Victor Stinner <victor.stinner@haypocalc.com> parents: 61847 diff changeset	1556 f.writer.write(data[0])
bf82c1d04e06 Issue #6268: More bugfixes about BOM, UTF-16 and UTF-32 Victor Stinner <victor.stinner@haypocalc.com> parents: 61847 diff changeset	1557 self.assertNotEquals(f.writer.tell(), 0)
bf82c1d04e06 Issue #6268: More bugfixes about BOM, UTF-16 and UTF-32 Victor Stinner <victor.stinner@haypocalc.com> parents: 61847 diff changeset	1558 f.writer.seek(0)
bf82c1d04e06 Issue #6268: More bugfixes about BOM, UTF-16 and UTF-32 Victor Stinner <victor.stinner@haypocalc.com> parents: 61847 diff changeset	1559 f.writer.write(data)
bf82c1d04e06 Issue #6268: More bugfixes about BOM, UTF-16 and UTF-32 Victor Stinner <victor.stinner@haypocalc.com> parents: 61847 diff changeset	1560 f.seek(0)
bf82c1d04e06 Issue #6268: More bugfixes about BOM, UTF-16 and UTF-32 Victor Stinner <victor.stinner@haypocalc.com> parents: 61847 diff changeset	1561 self.assertEquals(f.read(), data)
bf82c1d04e06 Issue #6268: More bugfixes about BOM, UTF-16 and UTF-32 Victor Stinner <victor.stinner@haypocalc.com> parents: 61847 diff changeset	1562
bf82c1d04e06 Issue #6268: More bugfixes about BOM, UTF-16 and UTF-32 Victor Stinner <victor.stinner@haypocalc.com> parents: 61847 diff changeset	1563 # Check that the BOM is not written after a seek() at a position
bf82c1d04e06 Issue #6268: More bugfixes about BOM, UTF-16 and UTF-32 Victor Stinner <victor.stinner@haypocalc.com> parents: 61847 diff changeset	1564 # different than the start
bf82c1d04e06 Issue #6268: More bugfixes about BOM, UTF-16 and UTF-32 Victor Stinner <victor.stinner@haypocalc.com> parents: 61847 diff changeset	1565 with codecs.open(test_support.TESTFN, 'w+', encoding=encoding) as f:
bf82c1d04e06 Issue #6268: More bugfixes about BOM, UTF-16 and UTF-32 Victor Stinner <victor.stinner@haypocalc.com> parents: 61847 diff changeset	1566 f.write(data)
bf82c1d04e06 Issue #6268: More bugfixes about BOM, UTF-16 and UTF-32 Victor Stinner <victor.stinner@haypocalc.com> parents: 61847 diff changeset	1567 f.seek(f.tell())
bf82c1d04e06 Issue #6268: More bugfixes about BOM, UTF-16 and UTF-32 Victor Stinner <victor.stinner@haypocalc.com> parents: 61847 diff changeset	1568 f.write(data)
bf82c1d04e06 Issue #6268: More bugfixes about BOM, UTF-16 and UTF-32 Victor Stinner <victor.stinner@haypocalc.com> parents: 61847 diff changeset	1569 f.seek(0)
bf82c1d04e06 Issue #6268: More bugfixes about BOM, UTF-16 and UTF-32 Victor Stinner <victor.stinner@haypocalc.com> parents: 61847 diff changeset	1570 self.assertEquals(f.read(), data * 2)
bf82c1d04e06 Issue #6268: More bugfixes about BOM, UTF-16 and UTF-32 Victor Stinner <victor.stinner@haypocalc.com> parents: 61847 diff changeset	1571
bf82c1d04e06 Issue #6268: More bugfixes about BOM, UTF-16 and UTF-32 Victor Stinner <victor.stinner@haypocalc.com> parents: 61847 diff changeset	1572 # (StreamWriter) Check that the BOM is not written after a seek()
bf82c1d04e06 Issue #6268: More bugfixes about BOM, UTF-16 and UTF-32 Victor Stinner <victor.stinner@haypocalc.com> parents: 61847 diff changeset	1573 # at a position different than the start
bf82c1d04e06 Issue #6268: More bugfixes about BOM, UTF-16 and UTF-32 Victor Stinner <victor.stinner@haypocalc.com> parents: 61847 diff changeset	1574 with codecs.open(test_support.TESTFN, 'w+', encoding=encoding) as f:
bf82c1d04e06 Issue #6268: More bugfixes about BOM, UTF-16 and UTF-32 Victor Stinner <victor.stinner@haypocalc.com> parents: 61847 diff changeset	1575 f.writer.write(data)
bf82c1d04e06 Issue #6268: More bugfixes about BOM, UTF-16 and UTF-32 Victor Stinner <victor.stinner@haypocalc.com> parents: 61847 diff changeset	1576 f.writer.seek(f.writer.tell())
bf82c1d04e06 Issue #6268: More bugfixes about BOM, UTF-16 and UTF-32 Victor Stinner <victor.stinner@haypocalc.com> parents: 61847 diff changeset	1577 f.writer.write(data)
bf82c1d04e06 Issue #6268: More bugfixes about BOM, UTF-16 and UTF-32 Victor Stinner <victor.stinner@haypocalc.com> parents: 61847 diff changeset	1578 f.seek(0)
bf82c1d04e06 Issue #6268: More bugfixes about BOM, UTF-16 and UTF-32 Victor Stinner <victor.stinner@haypocalc.com> parents: 61847 diff changeset	1579 self.assertEquals(f.read(), data * 2)
bf82c1d04e06 Issue #6268: More bugfixes about BOM, UTF-16 and UTF-32 Victor Stinner <victor.stinner@haypocalc.com> parents: 61847 diff changeset	1580
61847 99e9ed214aad Issue #6268: Fix seek() method of codecs.open(), don't read the BOM twice Victor Stinner <victor.stinner@haypocalc.com> parents: 60339 diff changeset	1581
20013 e3184e46303b Change the PyUnit-based tests to use the test_main() approach. This Fred Drake <fdrake@acm.org> parents: 18163 diff changeset	1582 def test_main():
28917 1741af48fd9f Combine the functionality of test_support.run_unittest() Walter Dörwald <walter@livinglogic.de> parents: 28774 diff changeset	1583 test_support.run_unittest(
42649 c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	1584 UTF32Test,
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	1585 UTF32LETest,
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	1586 UTF32BETest,
28917 1741af48fd9f Combine the functionality of test_support.run_unittest() Walter Dörwald <walter@livinglogic.de> parents: 28774 diff changeset	1587 UTF16Test,
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	1588 UTF16LETest,
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	1589 UTF16BETest,
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	1590 UTF8Test,
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	1591 UTF8SigTest,
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1592 UTF7Test,
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1593 UTF16ExTest,
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1594 ReadBufferTest,
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1595 CharBufferTest,
28917 1741af48fd9f Combine the functionality of test_support.run_unittest() Walter Dörwald <walter@livinglogic.de> parents: 28774 diff changeset	1596 EscapeDecodeTest,
1741af48fd9f Combine the functionality of test_support.run_unittest() Walter Dörwald <walter@livinglogic.de> parents: 28774 diff changeset	1597 RecodingTest,
1741af48fd9f Combine the functionality of test_support.run_unittest() Walter Dörwald <walter@livinglogic.de> parents: 28774 diff changeset	1598 PunycodeTest,
35484 4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	1599 UnicodeInternalTest,
31795 bf37429f6d9f Add test case for unicode(somestring, "idna"). Martin v. Löwis <martin@v.loewis.de> parents: 28917 diff changeset	1600 NameprepTest,
37719 cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1601 IDNACodecTest,
33782 a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1602 CodecsModuleTest,
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1603 StreamReaderTest,
40239 7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1604 EncodedFileTest,
35269 2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1605 Str2StrTest,
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1606 BasicUnicodeTest,
35702 22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1607 BasicStrTest,
40239 7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1608 CharmapTest,
7c4058681224 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1609 WithStmtTest,
61847 99e9ed214aad Issue #6268: Fix seek() method of codecs.open(), don't read the BOM twice Victor Stinner <victor.stinner@haypocalc.com> parents: 60339 diff changeset	1610 BomTest,
28917 1741af48fd9f Combine the functionality of test_support.run_unittest() Walter Dörwald <walter@livinglogic.de> parents: 28774 diff changeset	1611 )
20013 e3184e46303b Change the PyUnit-based tests to use the test_main() approach. This Fred Drake <fdrake@acm.org> parents: 18163 diff changeset	1612
e3184e46303b Change the PyUnit-based tests to use the test_main() approach. This Fred Drake <fdrake@acm.org> parents: 18163 diff changeset	1613
e3184e46303b Change the PyUnit-based tests to use the test_main() approach. This Fred Drake <fdrake@acm.org> parents: 18163 diff changeset	1614 if __name__ == "__main__":
e3184e46303b Change the PyUnit-based tests to use the test_main() approach. This Fred Drake <fdrake@acm.org> parents: 18163 diff changeset	1615 test_main()

Mercurial > cpython-lru_cache-weakref

annotate Lib/test/test_codecs.py @ 65035:87f4b32905df 2.7